Software engineering
From Wikipedia, the free encyclopedia.
Software engineering (SE) is the profession concerned with creating and maintaining software applications by applying computer science (CS), project management (PM), domain knowledge (DK), and other skills and technologies.
Software applications (including ATMs, compilers, databases, email, embedded systems, graphics, office suites, operating systems, robotics, video games, and the world wide web) embody social and economic value, by making people more productive, improving their quality of life, and enabling them to do things that would otherwise be impossible. SE technologies and practices (including databases, languages, libraries, patterns, platformss, processes, standards, and tools) help developers, by improving productivity and quality.
Software engineering is a community of people. The SE community includes 630,000 practitioners and educators in the U.S and 1,400,000(?) practitioners in the E.U, Asia, and elsewhere; and is about 60% the size of traditional engineering. American SE pioneers include Kent Beck, Barry Boehm, Fred Brooks, Watts Humphrey, and David Parnas.
There is considerable debate over whether software development should be considered a branch of traditional engineering, a branch of computer science, an independent scientific field, or a non-scientific craft. This article attempts to be neutral on this issue, but errs on the side of being independent to clarify the differences between fields.
As of 2004, in common parlance the term software engineering is used with at least three distinct meanings:
- As the usual contemporary term for the broad range of activities that was formerly called programming or systems analysis;
- As the broad term for the technical analysis of all aspects of the practice, as opposed to the theory of computer programming;
- As the term embodying the advocacy of a specific approach to computer programming, one that urges that it be treated as an engineering profession rather than an art or a craft, and advocates the codification of recommended practices in the form of software engineering methodologies.
Software engineering today
Software engineering matters
In the U.S., software drove about 1/4 of all increase in GDP during the 1990s (about $90 billion per year), and 1/6 of all productivity growth (efficiency within GDP) during the late 1990s (about $33 billion per year). Software engineering drove $1 trillion of economic and productivity growth over the last decade. See also software engineering economics.
Software engineering changes world culture, wherever people use computers. Email, the world-wide web, and instant messaging enable people to interact in new ways. Software lowers the cost and improves the quality of health-care, fire departments, and other important social services.
Successful projects where software engineering methods have been applied, include linux, the space shuttle software, and automatic teller machines. When it is cheaper to run a business or agency with software applications than without, businesses and agencies often invest in computers, software, and personnel.
Today's education
People from many different education backgrounds make important contributions to SE. The fraction of practitioners who earn computer science or software engineering degrees has been slowly rising. Today about 1/2 of all software engineers earn computer science or software engineering degrees. For comparison, about 3/4 of all traditional engineers earn engineering degrees.
Software: About half of all practitioners today have computer science degrees, which are the most relevant degrees that are widely available. A small, but growing, number of practitioners have software engineering degrees. Today in the U.S., about 2,000 universities offer computer science degrees and about 50 universities offer software engineering degrees. Most SE practitioners will earn computer science degrees for decades to come, though someday, this may change.
Domain: Some practitioners have degrees in application domains, bringing important domain knowledge and experience to projects. In MIS, some practitioners have business degrees. In embedded systems, some practitioners have electrical or computer engineering degrees, because embedded software often requires a detailed understanding of hardware. In medical software, some practitioners have medical or biology degrees.
Other: Some practitioners have mathematics, science, engineering, or other technical degrees. Some have philosophy, or other non-technical degrees. And, some have no degrees. Note that Barry Boehm earned degrees in mathematics and Edsger Dijkstra earned degrees in physics.
Graduate software engineering degrees have been available from dozens of universities for a decade or so. Undergraduate software engineering degrees are being established at many universities. A new international curriculum for undergraduate software engineering degrees is currently being defined by the CCSE.
Today's practice
Practitioners specialize in many roles in industry (analysts, developerss, testerss, technical support, managers) and academia (educators, researchers).
Most software engineers work as employees or contractors. Software engineers may work with a profit-seeking corporation (a business), a government agency (civilian or military), or a non-profit agency (a school or .org like Wikipedia). Some software engineers work for themselves as free agents.
Today's debates
Many debates are raging within SE. As software becomes more pervasive, we all recognize the need for better software, but we disagree on how.
Technologies and Practices: What is the best way to make more and better software? SEs advocate many different technologies and practices, with much disagreement. This debate has gone on for 60 years and may continue forever.
Identity: Is SE a branch of computer science, a branch of traditional engineering, or a field that stands on its own? Recently, software engineering has been finding its own identity and emerging as an important field. Yet, some advocate making SE a part of traditional engineering and others advocate keeping SE a part of computer science.
Professionalism: What will SEs do about professionalism, licensing, and ethics? Licensing is a polarizing issue. Some fiercely advocate it. Others staunchly oppose it.
Success: Is SE a success or a failure? Some look to the enormous economic growth and productivity gains enabled by software and claim that software engineering is a huge success. Others point to the ongoing problems with crashing operating systems and computer viruses and claim that software engineering has failed. How can we reconcile these points of view?
Current directions for software engineering
Aspect-oriented programming and agile methods are important emerging SE technologies and practices.
Aspects help programmers deal with ilities by providing tools to add or remove boilerplate code from many areas in the source code. Aspects describe how all objects or functions should behave in particular circumstances. For example, aspects can add debugging, logging, or locking control into all objects of particular types. Researchers are currently working to understand how to use aspects to design general-purpose code. Related concepts include generative programming and templates.
Agile Methods guide software development projects that evolve rapidly with changing expectations and competitive markets. The heavy, document-driven processes (like CMM and ISO 9000) are fading in importance. Some people believe that companies and agencies export many of the jobs that can be guided by heavy-weight processes. Related concepts include extreme programming and lean software development.
The Future of Software Engineering conference (FOSE) held at the ICSE 2000 documented the state of the art of SE in 2000 and listed many problems to be solved over the next decade. The Feyerabend project attempts to discover the future of software engineering by seeking and publishing innovative ideas.
Evolution
Software engineering has evolved steadily from its founding days in the 1940s until today in the 2000s. Applications have evolved continuously. The ongoing goal to improve technologies and practices, seeks to improve the productivity of practitioners and the quality of applications to users.
1945 to 1965: The origins
The term software engineering first was used in the late 1950s and early 1960s. Programmers have always known about civil, electrical, and computer engineering and debated what engineering might mean for software.
The NATO Science Committee sponsored two conferences on software engineering in 1968 (Garmisch, Germany) and 1969, which gave the field its initial boost. Many believe these conferences marked the official start of the profession.
1965 to 1985: The software crisis
Software engineering was spurred by the so-called software crisis of the 1960s, 1970s, and 1980s, which identified many of the problems of software development. Many software projects ran over budget and schedule. Some projects caused property damage. A few projects caused loss of life. Some used the term software crisis to refer to their inability to hire qualified programmers. The software crisis was originally defined in terms of productivity, but evolved to emphasize quality.
Cost and Budget Overruns: The OS/360 operating system was a classic example. This decade-long project from the 1960s and 1970s eventually produced one of the most complex software systems ever created. OS/360 was one of the first large (1000 programmer) software projects. Fred Brooks claims in Mythical Man Month that he made a multi-million dollar mistake by not developing a coherent architecture before starting development.
Property Damage: Software defects can cause property damage. Poor software security allows hackers to steal identities, costing time, money, and reputations. An expensive European Ariane rocket exploded because of software.
Life and Death: Software defects can kill. Some embedded systems used in radiotherapy machines failed so catastrophically that they administered lethal doses of radiation to patients.
Peter G. Neumann keeps a contemporary list of software problems and disasters at Computer Risks.
The software crisis has been slowly fizzling out, because it is unrealistic to remain in crisis mode for more than 20 years. SEs are accepting that the problems of SE are truly difficult and only hard work over many decades can solve them.
1985 to present: No silver bullet
For decades, solving the software crisis was paramount to researchers. Seemingly, they trumpeted every new technology and practice from the 1970s to the 1990s as a silver bullet to solve the software crisis.
Tools, discipline, formal methods, process, and professionalism were touted as silver bullets. Tools: Especially emphasised were tools. Structured programming, object-oriented programming, CASE tools, Ada, documentation, standards, and UML were touted as silver bullets. Discipline: Some pundits argued that the software crisis was due to the lack of discipline of programmers. Formal methods: Some believed that if formal engineering methodologies would be applied to software development, then production of software would become as predictable an industry as other branches of engineering. They advocated proving all programs correct. Process: Many advocated processes and methodologies like CMM. Professionalism: This led to work on a code of ethics, licenses, and professionalism.
In 1987, Fred Brooks published the No Silver Bullet article, arguing that no individual technology or practice would ever make a 10-fold improvement in productivity within 10 years.
Debate about silver bullets raged over the following decade. Advocates for Ada, components, and processes continued arguing for years that their favorite technology would be a silver bullet. Sceptics disagreed. Eventually, almost everyone accepted that no silver bullet would ever be found. Yet, claims about silver bullets pop up now and again, even today.
Some interpret no silver bullet to mean that SE failed. The search for a single key to success never worked. All known technologies and practices have only made incremental improvements to productivity and quality. Yet, there are no silver bullets for any other profession, either. Others interpret no silver bullet as proof that SE has finally matured and recognized that projects succeed due to hard work.
Major developments
There are a numbers of areas where the evolution of software engineering is notable.
Emergence as a profession: From the mid-1990s to the mid-2000s, software engineering emerged as a bona fide profession, to stand beside computer science and traditional engineering. See also software engineering professionalism.
Role of women: In the 1940s, 1950s, and 1960s, software was often written by women. Men often filled the higher prestige hardware engineering roles. Grace Hopper and many other unsung women filled many programming jobs during the first several decades of software engineering. Today, fewer women work in software engineering than in other professions. Saying that this is sexual discrimination is too simple, because it related directly to individual identity. In one sense, software engineering is the masculinization of programming. The roles women fill in SE continue evolving. Today, more women in software engineering fill the social roles of analysis, training, documentation, and management; and fewer do hard-core technical development.
Processes and Methodology: Processes and methodologies have become big parts of software engineering. Many practitioners resist process, which often treats them impersonally like machines, rather than like people.
Cost of hardware: The relative cost of software versus hardware has changed substantially over the last 50 years. When mainframes were expensive and required large support staffs, software projects could be expensive. Because powerful PCs are cheap, software projects must become cheaper, in comparison.
Comparing related fields
The relationships between software engineering and the fields of programming, computer science, and traditional engineering have been debated for decades. Software engineering resembles all of these fields, but important distinctions exist.
Comparing programming
Both programmers and software engineers work on all sizes of projects: small and large.
Programmers emphasize the task of writing code to produce working software applications, independent of budget and schedule.
Software engineering tries to encompass software projects more completely, including budget and schedule; fits in a large business context with relationships to marketing, sales, production, installation, training, support, and operations; and methods to construct large applications that individual programmers cannot write alone.
| Issue | Software Engineering | Programming |
|---|---|---|
| Scope | Relates programming to the final application | Emphasizes programming, independent of the application |
| Business context | Collaborate with others in business | Emphasizes individual work |
| Team size | Individuals to large teams | Emphasizes individuals |
| Number of Practitioners in U.S. | 680,000 | 530,000 |
Comparing computer science
Many compare software engineering to computer science and information science like they compare traditional engineering to physics and chemistry.
About half of all software engineers earn computer science degrees. Yet on the job, practitioners do applied software engineering, which differs from doing theoretical computer science.
| Issue | Software Engineering | Computer Science |
|---|---|---|
| Ideal | Constructing software applications for real-world use for today | Finding eternal truths about problems and algorithms for posterity |
| Results | Working applications (like office suites and video games) that deliver value to users. | Running time analysis, space analysis, and correctness of algorithms (like Shell sort) and analysis of problems (like travelling salesman problem) |
| Budgets and Schedules | Projects (like upgrading an office suite) have fixed budgets and schedules | Projects (like solving NP) have open-ended budgets and schedules |
| Change | Applications evolve as user needs and expectations evolve, and as SE technologies and practices evolve. | When computer science problems are solved, the solution will never change |
| Additional Skills | Domain knowledge | Mathematics |
| Notable Educators and Researchers | Barry Boehm, Fred Brooks, and David Parnas | Edsger Dijkstra, Donald Knuth, and Alan Turing |
| Notable Practitioners | John Backus, Dan Bricklin, Tim Berners-Lee, Linus Torvalds, Richard Stallman, Steve McConnell | Not applicable |
| Practitioners in U.S. | 680,000 | 25,000 |
| Practitioners in Rest of World | 1,400,000? | 50,000? |
Comparing traditional engineering
Software engineers aspire to build low-cost, reliable, safe products; much like traditional engineers do. Software engineers borrow many metaphors and techniques from traditional engineering disciplines, including requirements analysis, quality control, and project management techniques. Traditional engineers also borrow many tools and practices from software engineers. Yet, there are also many differences between SE and TE.
In the U.S., there are about 10 times as many software engineers as computer engineers. The software engineering community is about 60% as large as the traditional engineering community.
| Issue | Software Engineering | Traditional Engineering |
|---|---|---|
| Foundations | Based on computer science, information science, and discrete math. | Based on physics, chemistry, and calculus. |
| Cost | Compilers and computers are cheap, so software engineering and consulting are often more than half of the cost of a project. Minor software engineering cost-overruns can adversely affect the total project cost. | Construction and manufacturing costs are high, so traditional engineering may only be 15% of the cost of a project. Major engineering cost overruns may not affect the total project cost. |
| Replication | Replication (copying CDs or downloading files) is trivial. Most development effort goes into building new (unproven) or changing old designs and adding features. | Most development effort goes into replicating proven designs through manufacturing or construction. |
| Innovation | Software engineers often apply new and untested elements in software projects. | Traditional engineers often apply known and tested principles, and limit the untested innovations that goes into each product. |
| Duration | Software engineers emphasize projects that will live for years or decades. | Traditional engineers may solve long-ranged problems (bridges and dams) that endure for centuries. |
| Management Status | Few software engineers manage anyone, so they are rarely viewed as managers, except by themselves. | Many traditional engineers manage construction, manufacturing, or maintenance crews, so all engineers are widely viewed as managers. |
| Blame | Software engineers must blame themselves for project problems. | Traditional engineers can often blame construction, manufacturing, or maintenance crews for project problems. |
| Practitioners in U.S. | 611,900 software engineers | 1,157,020 total engineers 67,180 computer engineers |
| Age | Software engineering is about 50 years old. | Civil engineering is thousands of years old. |
Fighting over issues
The term engineering causes a lot of confusion.
With about 612,000 software engineers in the U.S., and many more around the world, there should be room for many different opinions and approaches. A consensus has yet to emerge.
Right to use the word engineering
The wrangling over the status of software engineering (between traditional engineers and computer scientists) can be interpreted as a fight over control of the word engineering. Traditional engineers question whether software engineers can legally use the term.
Traditional engineers (especially civil engineers and the NSPE) claim that they have special rights over the term engineering, and for anyone else to use it requires their approval. In the mid-1990s, the NSPE sued to prevent anyone from using the job title software engineering. The NSPE won their lawsuit in 48 states.
However, SE practitioners, educators, and researchers ignored the lawsuits and called themselves software engineers anyway. The U.S. Bureau of Labor Statistics uses the term software engineer, too. The term engineering is much older than any regulatory body, so many believe that traditional engineers have few rights to control the term.
The fields of data engineering, knowledge engineering, user interface engineering, and so on have similar concerns about the term engineering. Even smaller or newer fields of biological engineering, safety engineering, and corrosion engineering have these concerns.
Substance versus metaphor
Some believe that the name SE means that practitioners must also be traditional engineers. Others believe that engineering is only a metaphor that SEs should apply appropriately.
Substance: Those who define software engineering as a branch of traditional engineering often believe that SEs apply concepts from traditional engineering to software development. This means that software engineering student should study physics, chemistry, and calculus; practitioners should earn professional licenses; and so on. They believe engineering provides a structured, logical approach, and therefore, a stable final product.
Metaphor: Others are inspired by traditional engineering, but believe that software needs its own solutions. They believe that many traditional engineering concepts cannot apply, because software is fundamentally different than bridges and roads. For example, traditional engineers do not use compilers or linkers to build roads. They believe that students should study computer science and other useful topics, and they practictioner do not necessarily need licenses.
Relation to other terms
Prior to the mid-1990s, most software practitioners called themselves programmers or developers, regardless of their actual jobs. Many people prefer to call themselves software developer and programmer, because we widely agree what these terms mean, while software engineer is still being debated.
The term programmer has often been used as a pejorative term to refer to those who lacked the tools, skills, education, or ethics to write quality software. In response, many practitioners called themselves software engineers to escape the stigma attached to the word programmer. In many companies, the titles programmer and software developer were changed to software engineer, for many categories of programmers.
These terms cause confusion, because some denied any differences (arguing that everyone does essentially the same thing with software) while others use the terms to create a difference (because the terms mean completely different jobs).
Fighting over priorities
Everyone agrees that we want better software, but we disagree on priorities, approaches, and on what an individual should do in a specific circumstance. Everyone seems to advocate a different combination of the following issues. Proponents and methodologists advocate conflicting solutions and often heatedly debate their merits. All subfields mix the following priorities to varying degrees.
Management: Some advocate that software engineering is primarily about the management practices necessary to make reliable budgets and schedules. People at the Software Engineering Institute took this approach and created the CMM.
Formal methods: Some advocate applying rigorous mathematical analysis to computer programming, especially proofs of correctness. They believe that traditional engineering is carried out with mathematical rigor, while programming is an iterative, trial-and-error process. These advocates strive to make programming more rigorous.
Process: Some advocate that software engineers must follow step-by-step processes, much like assembly line workers. This inspired CMM, ISO 9000, RUP, SPICE, and other methods and processes.
Tools: Some advocate that software engineering means tools, especially CASE tools (like Unix tools and IDEss) that emphasize high-level architecture issues. Today's CASE tools emphasize UML.
Ethics: Some advocate that software engineering is mostly about codes of ethics and social responsibility. They sometimes argue that bugs are due to lapses of ethics.
Licenses: Some advocate defining software engineering in terms of professional licenses, like some traditional engineers have. The biggest advocates of this position are from Texas and Canada, where the state governments sponsor licenses for SEs.
Degrees: Some advocate defining SE by college degrees. Most professions have college degrees tailored to the needs of practitioners. Many graduate software engineering degrees are available and undergraduate degrees are becoming available. (Move to SE education?) Cost and Time versus Quality: Subfields like consumer applications are sensitive to time and cost. Subfields like military and medical applications are sensitive to quality.
Criticisms of software engineering
Critics argue that many of the foundations of software engineering are inherently flawed. The following paragraphs list many criticisms and responses. Note that many of these criticisms apply to other human activities including business and education.
- Managing Expectations
- Criticism: One key to successful software engineering projects is managing the customer's expectations to something that can be built and delivered. So, the field resembles marketing or sociology more than traditional engineering with its responsibilities to society at large and the perils of legal liability when they fail to protect the public interest.
- Response: Every profession manages expectations, including all branches of engineering. Moreover, responsibility to society means meeting the expectations of the general public, which is often a stakeholder.
- Poor Requirements
- Criticism: The requirements for most SE projects are incomplete or inconsistent. Some clients have little experience writing requirements. Other clients do not know what they want, and say "I'll know it when I see it" (IKIWISI). Even experienced clients who know exactly what they want may not precisely articulate their requirements. Clients often expect much more than they write in the requirements. And, requirement documents can describe applications that have no computable or practical solutions.
- Response: One response is to avoid all projects with poor requirements, which would avoid most projects. Another response is using agile development and rapid prototyping to clarify project goals and deliver value (the most important requirements) quickly. Embedded systems are special in that they fit inside products that are designed by other engineers, who can sometimes define the software requirements completely.
- Rising Complexity
- Criticism: Critics argue that the complexity of requirements and user expectations have only increased. The probability of failure increases with the size, scope and complexity of the project. Technologies and practices have consistently improved over the years, but the gap between what is expected and what is delivered has not improved.
- Response: Rising complexity actually shows the success of practitioners, because demand naturally follows supply. When clients demand more, it shows their belief that their demands will be supplied.
- Ongoing Change
- Criticism: Practitioners continually develop new technologies and practices and use them whenever possible. Critics argue that ongoing change proves that older technologies and practices were failures.
- Response: Many view ongoing change as proof that software engineering successfully learns and grows. Traditional engineers also continually develop new technologies and practices.
- Ongoing Failure
- Criticism: Critics argue that incomplete or poorly designed systems are still too common. The early disasters in the field did not prevent subsequent disasters.
- Response: No field that strives to do bigger and better projects has ever avoided all unintentional failures. Traditional engineers also have ongoing failures: automobiles kill 40,000 people every year in the U.S.; Three Mile Island, Chernobyl, and Bhopal Disaster harmed thousands; Space Shuttles Challenger and Columbia blew up; MTBE added to gasoline to reduce air pollution also contaminated drinking water. Although large, reliable software systems can be and have been constructed, software projects that fail during construction or in service are still too common. Ongoing failure is a problem for both traditional engineers and software engineers. Some claim that software engineering is already as predictable and reliable as many fields of engineering, such as space or biological engineering.
- Failure to Pinpoint Causes
- Criticism: Critics argue that unlike traditional engineers (who analyze failures, find precise causes, and set up guidelines to avoid them in the future), software engineers routinely fail to pinpoint causes of failure or delay precisely enough to avoid repeats in the future.
- Response: Debugging is the activity of pinpointing the cause of failures in applications. Process improvement includes the activity of pinpointing the cause of process problems. Software engineers routinely pinpoint causes and then use the results to create better languages, databases, processes, and applications.
- Nothing New
- Criticism: Critics argue that software engineers created nothing on their own, but merely use what computer scientists already know.
- Response: Software engineers developed optimizing compilers, make, cvs, extreme programming, scripting, and bug databases on their own, out of necessity. Regardless of who creates or improves technologies and practices, software engineers are bright enough to adopt the best ones.
- Anyone Can Do SE
- Criticism: Many bright people from other fields (engineers, scientists, business people) write spreadsheet templates or simulations, and eventually switch to writing large applications, and believe that they do software engineering. So, software engineering is not a special skill.
- Response: Software engineering is a skill that is refined through long learning and practice. Software engineers are the ones who get the necessary education and experience, and keep up with evolving technologies, practices, and applications. This is true of every skill.
- We Do Not Know What SE Is
- Criticism: SE does not yield to the standard ways of categorization, under traditional definitions of engineering (calculus and science). This claim is often made by critics who want to impose their own definitions on everyone else.
- Response: We know a lot about what software engineering is. Software engineering is grounded in SE technologies and practices, and applications; and in the community of SE practitioners. Of course, software engineers continue to disagree about many details.
- No Software Science
- Criticism: Civil, electrical, mechanical, and chemical engineering build on solid results from physics and chemistry. These results enable assembling complex systems in a principled and systematic way. No corresponding results are available for software.
- Response: Software engineering builds on solid results from computer science and information science. These results enable the building of very sophisticated software systems.
- No Theorems About People and Projects
- Criticism: No theorems explain why one software engineer is more productive than another. No theorems explain why some software projects succeed and others fail. Without such knowledge no engineering is possible.
- Response: No theorems explain why one mechanical engineer is more productive than another. No theorems explain why some some civil projects go over budget and fail, for example why the Big Dig in Boston went way over budget, or why 2 space shuttles blew up. No theorems about people or projects exist anywhere.
- Meaning of Success
- Criticism: According to a study by the Standish Group (ref?) in 2000, 28 percent of software projects were complete successes (meaning they were executed on time and on budget), and 23% failed outright.
- Response: Many engineering projects fail to live up to expectations: many bridges and buildings run over budget or schedule. Consider that 40% of all space shuttles have blown up and the rest have been out of service for years. Almost all custom housing projects runs over budget and schedule. Success rates for software projects are meaningless without context.
- Magic
- Criticism: When programmers work hard and the system works, customers frequently do not appreciate how difficult the task was.
- Response: This is true for every profession.
- Software Creation is Inherently Creative
- Criticism: In Zeppelins and Jet Planes [1], Philip Armour argues that software is executable knowledge, which is discovered in a creative process, where trial and error, learning, and the ability to challenge one's assumptions are important. The true "construction" phase of software development is already automated by compilers and linkers. The difficult aspects relate to gathering requirements and designing systems. These are more like a craft than a science or engineering task. So, the potential benefit of software engineering is limited to making it easier for developers to try out ideas, to discover errors earlier, and to give them information about the state of a software system.
- Response: Many software engineers use Agile Methods to embody the creative nature of software engineering and to foster learning.
Conclusion
The software engineering profession is big, important, successful (if imperfect), young (yet learning rapidly). SE practitioners continue improving their technologies and practices, and improving applications to better meet the needs of users and the public. SE is now emerging to stand proudly on its own, beside computer science and traditional engineering. See the list of software engineering topics for more details. See Important publications in software engineering for academic achievements in software engineering.