Category: Work

  • Whale Study (twice)

    Some time ago (last year) I did a whale study:

    I planned to make a cool Escheresk graphic with it (for a wedding gift to be sure) but I stopped working on the big picture graphic before it was done (whales do not fit together as nicely as butterflies, fish or birds).

    Recently, I was listening in on a phone conference for work. The meeting was quite productive, though my contribution was minimal (literally five minutes out of the hour and a half) though I did glean tidbits of important information that made listening to the call worth the time. As I was listening, I found myself bored. I grabbed some paper and made a real life origami whale study (with magnets on them so they could live on the refrigerator):

    Yes, the narwhal is my own genius variation on the whale pattern. By the conclusion of the call I had made two blue whales and two narwhals. By the end of the night I had added two turtles and two walruses. Sometimes I think that I have the best job ever.

  • Document Retrieval System (More than a New Toy)

    An essay for COMM 297R, 27 June 2011


    Some Background Information
    Five years ago, just after opening the doors to the public, our office operated just like most offices (especially for the time). As part of our provisioning, we had purchased an array of filing cabinets to suit our short term and long term document storage and retrieval needs. Within a few months of opening the store I, and others in the office staff, became weary of having to dig through those filing cabinets to find various pieces of paper that employees, vendors and customers requested. I also noted that we had purchased an All-In-One printer, fax, copier and scanner. We used all of the functions except scanning, we could not think of a use for it. Late one December evening, a use for the scanner came to mind: we could scan in all of these papers we are filing and then digitally retrieve them. This way they would never got lost, only rarely get misfiled, would be easier to store (digital files take up digital space, not filing cabinets full).

    The EDNA (Electronic Document Network Access) system was launched 15 December 2005 featuring all of the documents we had filed from our store opening to date. After proving the value of the system, the company put into place a short term document storage program (in case we realized in a couple of months that we had forgotten something) and then changed our document processing procedures to include EDNA. Later, financing documents were added into the mix under the name of FINSTEEN.

    Both EDNA and FINSTEEN served the company for nearly six years until there were simply too many documents (nearly 200,000 of them). Then the old search engine could not keep up and the system was put into an “ICU” until it could be revived. Though, we the completion of this project, the old EDNA Search engine was officially retired and the new DRS Search engine was put in its place.

    The Project
    After nearly six years of accessing documents electronically, the choice of going back to paper was not an option. The only reasonable solution was to load all the scanned file names into a database that would be capable of handling the nearly 200,000 files we currently had and tens of thousands more in the future. This would give us, in essence, what we had but in a faster, more stable platform.

    In addition to indexing the document names, I also wanted to use Optical Character Recognition (OCR) technology to enable full text search on those documents. OCR is a slowly developing, and unheralded technology that enables computers to convert scanned documents into editable text. Though OCR has limited applications, I felt that in this particular instance our clearly printed documents and standardized formats would be good candidates for the technology. Once the OCR was indexed, we would be able to more easily find lost and mislabeled documents.

    Going Shopping
    Knowing that it is often less expensive (and frequently better quality) to find an already developed project that we can purchase rather than develop in house, I started searching for a suitable, existing search engine. The search did not go very well. All but one of the engines I found—that would be affordable—could index the document names, but none of them would index the contents. Not indexing the documents’ contents would nullify any benefit offered by the OCR process. Though using OCR is not a critical issue, it is one that I feel will strongly benefit us in future ways that we, as yet, do not know.

    The one I did find that would expose the OCR data in the search required a specialized platform and additional programming. In order to maintain simplicity, deploying a specialized platform was a detractor. I have a strong preference to fit any new search engine into our existing systems (systems that are well suited to the task), as opposed to building a specialized platform.

    After a week of searching and testing a variety of search engines, I concluded that we would need to custom build our search engine in order to get the features and flexibility we wanted.

    Solving the OCR Issue
    The first break through in the project was finding Google’s Tesseract software. Tesseract is a free, open source, software that reads TIF files and outputs a text file. While other software could do the same thing, Tesseract was the only software (that did not cost thousands of dollars) that could do the file reading automatically from a script. Every other piece of software required user interaction, a level of overhead that would negate the benefit of the new process.

    The ability to run the OCR from a script sacrificed a degree of reliability (though it is still very accurate) but allowed us to benefit from the OCR data without adding any extra labor costs. It also allowed us to keep the entire process after the manual file naming automated.

    DRS Search
    I found programming the actual search engine to be a fairly easy and straight forward process. I found several online tutorials that guided through the process and even taught me some interesting programming techniques to close obvious security loop holes and prevent simple data breaches. I was grateful for these tutorials because I am not versed enough in PHP to know what security vulnerabilities I have.

    The entire search engine is built into a single, elegant web page that can be accessed directly or, more commonly, accessed through our company intranet site. The previous EDNA Search contained four separate files, brought together at runtime to appear as a single webpage to the end user (though search results were returned in a separate page). The new single file format made the new search engine a lot easier to code and, more importantly, to troubleshoot. For the end user, the single page also made using basic web browser functionality such as the Back and Forward buttons work as expected. This has reduced the frustration of clicking Back and losing all the previous search results.

    Learning SQL
    As I was developing the DRS Search page, I started by mimicking the basic functionality of the EDNA Search page. One of the key features of the old search page was selecting the type of files the user wanted to search: sales or financing. Late one night, as I was ponder the user interface—more specifically, how to cram all the options (sales/financing, filename/OCR data and filename) into the search page while keeping it clean and respectable— it struck me: sales documents are always noted by their sale order number (e.g.” 10542230_so20110623.tif”) and financing documents are always noted by the customer name (e.g. “Smith, John wf20110623.tif”). This simple difference meant that whenever a user entered a numerical search they wanted a sales document and when they entered an alphabetical search they wanted a financing document. I talked with Isabel who confirmed by understanding and I set about programming the search engine to search both the sales and the financing documents at the same time.

    Initially when I set up the search database I had planned to keep the indexes of the financing documents separate from the sales documents. Though not for a particular reason, I simply thought that there might be a reason later on and that it would be much easier to combine the two database tables at a later date than to try to separate them. This caused me great frustration in trying to get the search engine to search both tables at the same time, with combined results, sorted alphabetically.

    At first I thought I would just “fake” it by search the sales and then the financing. While this was easiest to do, it double the lines of code for the search and created two separate, alphabetically sorted lists that would be confusing if there happened to be search results in both tables. (The previous system would run separate search and group the results by area, so sales documents were all grouped together and financing were all grouped together under respective headings.)

    Instead of giving up and taking the easy way, I decided to actually learn some SQL (the database language). The little SQL I already knew was very limited and centered around data extraction; my knowledge of the language was so limited that I would only risk changing two parts of the query, hoping I could transfer the data in Excel. I learned SQL. Not enough to be proficient at it, but enough to now understand how the queries I use actually work.

    In my learning of SQL, I learned two new words: LIKE and UNION. In this particular problem, UNION was the magic word. It took the two tables, temporarily combined them and allowed a search to be run against them and for the results to be returned into a single, alphabetically sorted result. No need to fake, I learned the real deal.

    Prior to my delving into SQL, I only knew the WHERE IS statement. This statement is the silver medal in search: WHERE you find something that IS equal to this thing. In order to make the OCR data useful, I needed something more powerful. To WHERE IS, the vast text collect by the OCR process would be nearly impossible to match against, thus making it useless.

    WHERE LIKE changes all of that. Instead of looking for an exact match, the search is looking for a similar match. To humans, navy blue IS blue but to the computer navy blue is LIKE blue. Changing this simple word opened up the entire range of OCR data to free form search.

    Giving Up
    While the DRS Search page was difficult for me to program, the actual indexer proved impossible. I spent more than a week solid trying to learn, copy and mimic techniques in hopes of crafting my own piece of software that could read the contents of the OCR files and add them into the database. After a week of trying I gave up.

    Daniel: Adam, I have a problem.
    Adam: What is it?
    Daniel: I need some PHP code that can scan a directory and put the filename and contents of each text file into a database. I have tried for more than a week to do this. I just cannot get it.
    Adam: Have you tried the GetContents command?
    Daniel: I just want it done. Can you do it?
    Adam: Yeah, I can have it to you by tomorrow.
    Daniel: You are a lifesaver. Send me the bill when it is done.

    Two hours later the finished code was in my inbox and the last, and most difficult, piece of the new search engine was in my possession. I felt like a dirty politician who just gave up winning the elections the fair way, but I also felt incredibly relieved: the great burden of the programming that very complex code was done. Adam’s code provided the perfect amount of flexibility so that I could easily fit into the project.

    Circular Referencing
    The final challenge in the project was how to automate the process of getting the sources files from the various offsite locations, running them through the OCR, indexing them in the database, and putting them in their final resting place.

    I started with existing functional infrastructure used by the EDNA system, the EDNA Trickle Transfer (ENDA TT). EDNA TT was perfect for moving the files from the remote locations to a central one, and I was able to add the code needed to automate the OCR process. This evidenced a major deficiency in the process: EDNA TT ran every hour of the hour. The time it took the OCR process to run varied from several minutes to over an hour depending on the number of files in the batch. This meant that the old timer mechanism would not work. I thought of running daily batches instead of hours batches knowing that that would guarantee sufficient time, but such would mean users would have to wait an entire work day before accessing newly added files, an unacceptable wait.

    A solution came one day as my roommate was describing the how the human body can detect an imbalance and releases a hormone to trigger a cascade of hormones to bring the body back into balance. Then it struck me, I could have the OCR process trigger the indexing process once it was done, once the indexer completed it could then wait for a period of time (I chose two hours) before triggering the OCR process and completing the loop. This circular process has proven to be a perfect and reliable system for giving each process enough time to complete with overlapping.

  • The Storyteller

    This is an autobiographical essay in response to the ever so frequent question: what do you do at work?

    The warehouse, plain and bland, stands just off the road like a giant fortress. Like all good fortresses, it is not the outside that attracts visitors. Rather, it is the treasure that lies inside it. Outside, the warehouse looks drab with its towering light brown walls and black trimmings, the office door doesn’t help much: “Employees Only” the small placard warns those who would open the foreboding black door.

    On the other side of the door things change little. The interior walls are a lighter shade of brown, almost taupe, and while there is no decorative trim, the walls are mostly empty. In the office, the workers are chatting; sometimes on the phone, sometimes to each other. Such is the humble work place of Daniel. Although he often works from any of the company’s other four locations, this is his chosen sanctuary to work his wonders.

    Although the outside is bleak, the inside office is obviously better suited to his creative thinking. The chatter of the customer service representatives provide a gentle background noise and people to occasionally socialize with and bounce ideas off of. But it is the massive twenty foot white board that makes the office ideal. “This,” Daniel says referring to the white board, “this is the only white board we have that is big enough to unload my brain onto.” Currently the white board is covered with scribbles, notes and a mass of lines and boxes. He uses the white board to stage his “stories”.

    Isabel, the Customer Service Manager, refers to him as an “Information God”, a title that makes Daniel laugh—he prefers the title of “Storyteller”. Isabel tells of when he first set up the company’s Customer Service surveys. The Customer Service Office had been hoping, at best, for a spreadsheet that tallied the results and were worried mostly about the ease of gathering the data. Instead of the basic spreadsheet they requested, Daniel delivered what he calls “a beautifully matriculated masterpiece of storytelling.” The survey system provides a friendly entry system for collecting the survey information and a full set of graphs and charts to explain the results, none of which require any technical expertise. “Like any good story, the mechanics are there, but they’re hidden,” Daniel explains.

    This is how most of his “stories” work: they feature quick and easy access to the data through charts, graphs, buttons to automatically retrieve up-to-date data , and “smart, dumb” text—complex formulas that output different texts based on the data in the spreadsheet so the responses look smart, but really aren’t.

    The ease with which people can use his spreadsheets has made them popular in the company, but he says that people should thank his boss for that, not him. “I like a good graph or chart, but the real data is where the best stories are found,” he says, “it was my boss who insisted that I make the stories in the data easier to see.”

    He sits hunched over his meticulously clean desk. The wood surface has on it a grand total of five objects: his phone and keys, lying side by side on the left side on the desk, a laptop, a second monitor and a wireless mouse. While his physical desk is nearly empty, his two screens are not. The laptop is cluttered with his email and various informational pages and the second monitor, hooked up as an extension of the laptop, is filled with a massive spreadsheet. It is this very spreadsheet that he often thinks of when people ask him if he knows Microsoft Excel. “Know it,” he says, “I live in it.”

    While he might joke about his knowledge of Excel, compared to most people, he does live in it. And like a monk left to himself to delve into the depths of sacred works, he knows Excel extremely well. “I often laugh when people ask if Excel can do certain things,” he laughs at this thought. “I usually tell them, ‘just tell me what you want and I’ll make Excel do it for you’.”

    This particular day he is working on one of his most complex “stories”: the company payroll. The numerous windows displayed across his screens are all critical story elements. They are part of a massive revision that he recently released. He explains that each screen has a function and purpose, and while they all together may seem overwhelming, no one else ever sees them all together.

    While some might think it an incredible feat, to him it is little more than a documentary. “No one does all the work,” he explains. Some poor soul digs to find random statistics that will be quoted in the voice overs. Another poor soul does the preliminary location and people research. The lucky host goes out and shoots footage with the camera crew. Then yet another host of people come in to cut the footage together and scale the production to the correct level.

    Payroll, for him, is no different. The new version—called Blackfin after ocean tuna—has simple, little spreadsheets and databases that different departments enter in little bits of information. At payroll time a single, bright green button, labeled “Extract All”, is pressed and through Excel’s magic all the little spreadsheets and databases are rounded up and processed to the familiar, but ever changing, story of payroll.

    Later, after he is done patching the payroll file, he walks to the back of the warehouse and rummages through a mess of old parts. The parts were recently purchased as a lump-sum from a business that was closing. He’s not looking for anything in particular, but more just wants to get away. His excursions into the warehouse are usually fruitless themselves, but they allow Daniel to refocus his mind. “I never know what I’ll find back here,” he says. Moments later he coos, “Ooh, these belong in the IT room,” he says with an elevated pitch as he wraps a bundle of network cables around his neck. Obviously, he’s done this before.

    Satisfied with today’s find he heads back towards the office. He doesn’t get far before he stops again, this time to pick up a rolling office chair. “The guys keep stealing chairs from the office for their lunchroom,” he explains. Instead of pushing the chair to the office, he sits in it, grabs the arm rests, carefully aims the chair and then, with cables still dangling from his neck, gives a swift kick and sends himself hurtling down the aisle between dining chairs and bedroom mirrors. The noise of the rolling chair can be heard throughout the whole warehouse, just one example of his creative eccentric nature.

    Back in the office, after dropping off the cables and the chair, Daniel examines his white board. He crosses some items off the board, and then taps his marker against the board. The next item on his list is to put up a reminder about upcoming network changes. He sits back down at his desk and brings up the company intranet site; he considers the site to be one of his greatest stories. “Everyone in the company uses it every day,” he says with pride in his voice. He starts to tell the story of making the site such a success, and then pauses as he looks around the office. “I’ve never told them the secret of my success.” After a few seconds of typing, he reviews the reminder and with a satisfactory nod posts it online.

    Those secrets are some of his rarest stories. In fact, he boasts, no one person has heard them all, and he intends to keep it that way. “If anyone knew that all I do every day is tell stories,” Daniel says with a smile, “well, they might find a way to live without me.” With that, he goes back to work: reviewing data, looking for connections and recording the stories he finds.

  • The Curse of Assumed Common Goals

    I have a particularly fond memory of my younger brother (though, I must admit that my memories are not the clearest on this topic and I might have embellished parts). It was back when we were all younger. The family, all except my brother, was all gathered in the living room to watch a movie or some such. As we were enjoying our peace my brother came suddenly bursting into the room yelling, “I got it, I got it!” His entrance made such a shock that we all shifted our attached to him and the “it” he had “got”. My mother, in a caring tone, congratulated him and asked if she could see “it”. He was so proud he popped open his hand and showed her. To her great horror, and the family’s astonishment, there, lying in his tiny palm was my mother’s beta fish. It was beyond all help as my younger brother had, in his great excitement, neglected to handle the fish with care and it had been crushed to death while in transit from the upstairs bedroom (where the fish had no doubt been sleeping) down to the living room.

    Though there are many humorous spins and analogies that I could make with this story, there is one in particular that I wish to make: there can be grave danger in assumed common goals. My mother’s goal was to keep the fish alive while my brother thought the goal was to catch the pesky fish. In this case, the assumed common goal (that really was not common) led to the death of a fish, but in many cases the assumed common goals can lead to much greater problems.

    On some level we all know and understand this, but even in the most basic of assumptions we all too quickly dismiss this truth. Instead of soliciting our goals, we assume that everyone, or at least the people in the immediate vicinity of our lives, have the same basic beliefs. Then we are faced with shock, dismay or disappointment when we find out that what we thought was a universal goal was really only a personal goal.

    This curse of assumed common goals is pervasively around us in our daily endeavors and projects. Think of the last time you were driving with a young child in the car. Your primary goal was likely to travel safely and you likely assumed that everyone else on the road had the same goal. This was probably not the case. Instead, most people probably had a primary goal of getting to their destination quickly, or to find out where their friends are before they pass them.

    While driving safely on the road might not be every driver’s primary goal (though it should be) it is still a high goal for most drivers. A better illustration might be a work place venue: collaborating to reduce costs across the company. There are many ways to go about the process of reducing cost, but I will focus on two: by eliminating jobs and rearranging the company accordingly; and by streamlining processes and eliminating positions that are no longer needed. Both methods will result in the end reduction of overall costs for the company, but they each have very different focuses.

    The process of eliminating positions is one that is likely to have a primary goal of seeing what the company can live without. Do we need someone special to clean up, or can we convince other employees to do the work? Do we need someone to oversee these workers, or can we assign them to a different manager and expect less supervision? With this goal in mind, reducing costs is much like dieting with a very definite list of things that you can and cannot do without.

    Consider working under the assumption that the other departments in the company were also operating under this same paradigm. Imagine the shock and horror you would experience when you found out that another department was in fact binging instead of dieting. They have been spending money on projects that, in the dieting mind set, should have been cut. This horror is likely to come because the offending department is operating under a different way of reducing costs.

    The process of streamlining is one that is likely to have a primary goal of automating as much as possible. Is there any way to digitize this data without manually inputting it? Can we make this paperwork flow smoother so it takes less time? Can we create self-help forums for employees to handle basic issues by themselves? With this goal in mind, reducing costs is much like exercising with a list of practices that will allow you to stay trim without foregoing all of the good stuff.

    While both of these methods can produce the same desired results, reduced costs, they have very different methods of achieving those results along with different operational positions (the basis for which decision are made) and the end results that each gains. It is while considering these minor differences that the real differences between the common goals becomes pronounced and, as they continue, begin to clash.

    The elimination of positions (dieting) is operating from a position of retreat or withdrawal. It says, “We have overreached and done too much to possibly be able to maintain our position so we must cut back even if that means that we give up some good stuff.” It is a defensive reaction to the environment and sends a clear signal that the only way to survive is to cut back.

    The irony of these two styles is that both are equally valid expressions of the same goal, it is entirely possible for managers in both departments to do extensive work (even while working together) to reduce costs and only towards the end of the process realize that the other has been going about the process entirely differently than they have. The incorrect assumption of a common goal is not likely to bother the streamlining manager, but is likely to cause quite a stir for the eliminating manager because he has been sacrificing and “going without” for the benefit of the company only to find out that others have been enjoying themselves and still saving money simply by “exercising” a little.

    The two styles will clash dramatically when the cost savings proposals are put forth. The elimination position will present a plan that requires cut equal to the savings. It will present no long term incentive for taking action but will instead focus on how the cuts will benefit the company now. As operations improve, this position will insist that the eliminated workers are restored in order to keep up with the new demand.

    The streamlining position will present a radically different plan that requires upfront investment in order to improve systems. Upon the completion of the system improvements, efficiencies will be realized such that positions can be eliminated or reallocated. There is no short term incentive for taking action (in fact, spending money to make money is often considered a disincentive) but the plan will instead focus on bringing savings to the company in the long run. As operations improve, this position will not insist on a directly proportional increase of workers because the improved systems can handle more throughput with less workers.

    In the end, both managers will have put forth effort to reduce costs, both managers will have action plans that will save money for the company and both managers will be in alignment with the overarching company goal of reducing costs. They will, however, have plans that look nothing alike and the end results will be dramatically different all because they did not share the same common goal.

  • The Virtues of Closed Versus Open

    I keep hearing HTML 5 this and HTML 5 that. The funny part, to me, is that while people are making stuff in HTML 5, it is not a standard yet. The W3C (World Wide Web Consortium, the people who are in charge of defining HTML) is taking forever to settle the finer points of how the new web standard should work. I am not exaggerating too much when I say that. Consider that HTML 3 (January 1997) was released a little more than a year after HTML 2 (November 1995) had been published. HTML 4 was released less than a year after that (December 1997). In the year 2000, the W3C gave us HTML 4.01 and XHTML 1.1 to help make programmers lives easier when using new media content (notice that was when the internet started looking and feeling more like it does today). The newest version, HTML 5 is designed to make the programmers lives even easier and thus make the internet even more amazing of a place. Work on the new version started in 2008, now three years later we are still waiting. A “working draft” has been published meaning, “We think it will look something like this, so start programming but do not be shocked if we change it on you later on.”
    While the “draft” has been published and programmers have been slowly working with it, many companies such as Mozilla (makers of Fire Fox), Apple and Google are pretending that the draft is gold (a programming term that means, “this is the final release for production”) and are running hard to make the browsers play well with the new almost standard. The problem with this is that they are almost standards, but not quite, and many of the particulars have not been settled which means that different browsers are handling things differently. For example, drop shadows in Fire Fox work differently enough from Safari and Chrome that web programmers (the people that these standards are supposed to help) have to right triple the code in order to use the cool new effect (once the HTML 5 way, once the Fire Fox way and once the Safari way). And triple is not even counting all the extra coding that has to be for Internet Explorer that does not play nice with anyone else.
    Back to the main point of this post: the main reason for this splintering is because HTML 5 has taken longer to develop and get out the door than any prior version of the standard. You may ask why this is the case. Surely this is not because HTML 5 is so much technically superior that a lot of extra effort has to be taken to develop it, a lot of the technical coolness is in making the code flow freer. It certainly is not because the W3C is waiting to see how that world will continue to evolve, HTML 6 can take care of that. No, the reason that it is taking so long to get HTML 5 from draft to standard is because the big boys (that were not there, or at least were not as big, for previous versions) spend too much time fighting amongst themselves that they cannot decide anything. Google wants to support their WebM video format because it is free and open. Apple wants to use their H.264 MP4 format because they think it is the best (and they happen to control the H.264 standard), but no one else wants to use it because Apple often does crazy things that throws everyone for a loop. (For example, the whole Photoshop on the Mac issue, a summary follows.)

    In case you missed it: When Mac OS X was released in March 2001, Apple caused a major stir amongst computer programmers. OS X was the first major operating system to severely break backward compatibility. Most other systems (namely Microsoft Windows) typically took great pains to remain reverse compatible while adding new features. But Apple, in a very “Steve Jobsish” way, suddenly said, “Um, world, we are going to change our core programming language in our next release (later that year) so start rewriting your programs because your old programs won’t work as well.” This threw everyone for a loop and the computer world was full of a lot of grumbling, mostly over the incredible expense of rewriting an application in a new language and on such short notice. Because of this last minute switch, there were few applications ready for OS X when it was released. Most of the programs that were available were from Apple itself. Adobe, creators of the extremely popular Photoshop software (and in many ways a competitor with Apple’s popular Final Cut Pro and Garage Band software) rolled their eyes and released a compatible version of Photoshop a year later. Fast forward several years to October 2007 when, concurrent with the release of a new version of OS X, Apple announced that once again, they were changing the foundational programming of the operating system, this time they would end support for 32-bit programs. Microsoft made similar announcements concerning Windows Vista, Windows 7 and Windows 8. Adobe was ready with an updated version of Photoshop that moved in Microsoft’s desired direction shortly after the launch of Windows Vista and 7 (Windows 8 has yet to be launched or even finalized, but is supposed to be the first version of Windows that only support 64-bit, this means that Microsoft gave the world more than 5 years advanced notice for what Apple gave a few month’s notice). Does Adobe love Microsoft so much more than Apple? Maybe. Does Adobe love Windows users so much more than Mac users? Not really, especially considering that Adobe has a strong Mac customer base and the Windows customer base just recently caught up. Why then was Adobe ready for Microsoft’s changes and not Apple’s? The difference is in the subtleties of the timings. Apple announced their dramatic change months ahead of the release of the new OS (mid-2007), giving the company little time to do anything but sneeze. Microsoft announced their dramatic change in 2003 and 2005 for Windows Vista, 2005 and 2007 for Windows 7 and 2007 and 2009 for Windows 8. The difference is that Apple has a history of surprising everyone with big changes that are happening right away while Microsoft (and most other large companies like IBM, Oracle, Sun Microsystems and Google) tend to give other companies a lot of notice for big changes. By the way, Adobe didn’t release an updated version of Photoshop until April 2010, two and a half years later. The lengthy delay and the “nasty-gram” sent just after Apple’s announcement were meant to say, “knock it off or we will stop playing nice.” Apple responded by not supporting Flash on the iPhone and then banned applications made using Adobe Flash tools (until the government started eying them for anti-trust law violations).

    Adobe, who also sits on the W3C, wants to continue to push Flash to the forefront tauting that with Flash, users do not have to worry about the format, the people building the site do. While Microsoft is fighting for their favorite WMV format, which stands for Windows Media Video and thus no one wants to touch it even with a long stick. All this, and we have not even touched to the “groups” yet. Consider the Motion Picture Experts Group. Heard of them? Probably not directly, but they are the people that invented, and control, the MPEG standards including the incredibly popular and world famous MP3 format, MPEG 1 and 2 formats (used for DVDs and satellite TV respectively) and MP4 which is quickly replacing the previously mentioned formats. Each iteration allows for more compact files (read: faster download times) without sacrificing quality (but your computer has to do a lot of extra work). All of these companies and groups are arguing over which formats and standard must be support, and this is just for video. I am not even going to touch audio or graphics.
    All of this lengthy process of deciding on which video codec to use as the new web standard, a standard that within the next couple of years will need to be revised and at that point, the most current and popular video codec can be added. But, what we see here is all of these companies realize the extreme value in being the “chosen” codec until the new standard can be published.
    As a direct competitor to HTML 5 (at least in the mind of Apple Inc.,’s Steve Job) we have Adobe Flash. Originally released in 1995 as Future Splash by Smartsketch, Flash has been bought out twice: once by Macromedia, in 1996, and again by Adobe in 2005. According to Adobe’s computer census, Flash enjoys a nearly perfect market penetration, with recent (being the current and previous version) versions on 99% of computers on the internet. That is a huge market penetration, of which I can think of no other non-governmental product (i.e. electricity, water and sewer) that comes close. In more than 15 years, developers and consumers (mostly forced by developers) have consistently chosen Flash so strongly that even in an era of cutthroat business practices including the great Dot Com Bubble burst of 2000, Flash has expanded adoption in order to capture the entire market and has kept pace to maintain its market share. (In fairness, Flash was aided by being included as a default plugin with Microsoft Internet Explorer for a time.)
    Now, let us consider what Flash has done in their 17 years of existence. Flash came from humble beginnings, as rightly it should. It was originally developed as a pen drawing tool that was built out to include vector animation for the then fledgling internet (very few people had internet access in those days). The few people who did have internet had very limited bandwidth. This bandwidth limitation kept most web pages confined to simple text and  basic backgrounds (in HTML, colors are set with a simple hex code such as #000000 for black or #FFFFFF for white, hex code takes up a lot less bandwidth than graphics). While some graphic formats were available for animation, such of Compuserve’s GIF, which included transparency too), these graphics were limited by color restrictions (GIFs can show 256 colors, 255 if one color is transparent), a lack of support (the Motion Picture Experts Group’s MPG format still is not well support in browser), a difficulty to produce (no easy-to-use MPG animation tools were around in 1995), and if developers had been able to overcome all of the aforementioned obstacles they were likely to have ended up with a “huge” file that, though small by today’s standards, would have taken web page viewers hours to download. All of this meant that the not so wonderful animation was not likely to be viewed by anyone and thus was billed as wasted effort.
    The introduction of Flash changed all that. Flash used vector graphics. While not the best for every application—people pictures do not look too good in them—they are excellent for general shapes and lines. In computer speak, shapes and lines are simply geometric formulas that are easy to store, easy to transmit and easy to render.  Using HTML’s simple hex color codes made coloring them really easy too. Finally, because vector graphics are simple calculations, they are really easy to make bigger and smaller: for twice the size, simply have the computer multiply by 2 (or whatever other multiplier you want). As vectors get larger they maintain their crisp, sharp look, unlike the alternative graphic formats.
    Macromedia produced two components to Flash: the authoring tool and the plugin which is used to view Flash. The authoring tool meant that designers could produce their own animations without having to wait for programmers or video producers to work with them, a huge benefit for the designer. While the authoring tool costs money, developers could show their work to the world without worrying about their consumers having the right plugin: if a consumer did not have Flash, or the correct version, they could quickly download it for free. This setup led to Flash’s strong adoption throughout the early internet among both producers and consumers.
    After releasing the first version of Flash, Macromedia continued to add features that were of great benefit to developers, features that, in effect, severed the need pervasive trend (at least, to an ever increasing larger degree) of a programmer to work with the designer to make compelling content. While this may not seem significant, consider the two different worlds that programmers and designers come from and how difficult it can be to for one to communicate the vision of what they are trying to do in a way that the other can readily understand and accept.
    In 1997, two years after Flash’s first release, Macromedia added a host of new features and laid the foundation that producers would be using for years to come. The next year, in 1998, Macromedia sensed that the relatively new MP3 format would become a powerful market force and added support for playing the new format from within Flash. This greatly expanded the capabilities of Flash. Prior to that point, Flash could only handle WAV and MIDI files. WAV files are actual recordings of real sound that generally include a full audio spectrum, but also includes a full file size (good quality wave audio is about 10mb a minute, far too large for the still restrictive bandwidth to allow, especially when compared to MP3’s 1mb a minute). MIDI is synthesized sounds and music; while the files are really small, they are also very limited in quality and sound more like a Queen song that the London Philharmonic Orchestra. The following year, streaming MP3 support was added allowing consumers to start listening to sounds and music without downloading the whole file.
    The years 2000 and 2002 saw the introduction of Actionscript and video support respectively. The addition of Actionscript welcomed programmers back into Macromedia’s internet development arena by providing them a programmatic interface in which they could further enhance Flash animations by giving them custom logic sets (to do things that Macromedia had not imagined Flash doing) and allowed for dynamic content. No longer was the developer limited to text, words, video and information that was available at publication, using Actionscript, content could be added and updated on the fly without republishing the Flash project. The introduction of video not only extended the features of Flash, but also offered a solution to a prevalent problem across the maturing internet: producers, developers and consumer could not decide on a video production tool and plugin. Adding it to Flash made the choice simple, especially when their then more than 90% market penetration was considered. Almost everyone on the internet already had Flash, so why not use it. Additionally, Flash allow the developers to make custom interfaces so the video frame looked and felt like the rest of the site; the producers could protect their video against download and theft (though, later tools would emerge to work around this); customers often need do nothing, the magical plugin they downloaded in the past would magically play the video they wanted to watch.
    The 2003 release of Flash completed the foundational process of turning Flash into a truly ubiquitous platform both on and off the internet. This release included Actionscript 2, extending and deepening Flash’s programmatic abilities over the previous version that made advanced programming functions such as database interaction and dynamic content rendering more powerful and easier for the programmers; extended support for popular graphics formats, Adobe Illustrator files (a popular and powerful vector graphics program, think of it as Photoshop for vectors); inclusion in the popular (especially amongst business users) Adobe Acrobat; and charting features for business users. By this time, Flash had already achieved nearly universal distribution across internet connected computers and these additional features strengthened its position amongst both business users (who want to ensure the secure, as in difficult to steal, and reliable distribution of corporate information, and consumers who had begun to rely on the internet as not just a source of information, but a source of rich and engaging media.
    After Adobe’s acquisition of Macromedia in 2005, the development of Flash continued including the release of Actionscript 3 (continuing the strengthening of the developers’ side of Flash while securing Flash by disallowing Flash to interact with the host without the users knowledge), and the inclusion of upcoming video encoding standards including H.264 and MP4, and tighter integration with Adobe’s other development environments such as Adobe Illustrator and Photoshop.
    In 2009, Adobe delivered on their promised goals announced when Adobe bought Macromedia: Flash became an open standard. By publishing the specifications for Flash, Adobe knocked down one of the last barriers in adoption of Flash (not that very many people were actively opposed to it). Now everyone could see exactly what made Flash tick and could even build their own Flash plugins and development tools if they so desired. While some effort has been made to create a less expensive alternative to Adobe Flash (the development program) the comparative difficulty of programming a suitable development environment to the low cost of Adobe Flash has proved too difficult to overcome. Adobe has maintained a competitively priced product by separating the more advance Adobe Flash features into Adobe Flash Professional, leaving a lower featured and prices, though still adequate, Adobe Flash on the market and by including Adobe Flash in their Creative Suite bundles.
    As to custom plugins, no one really trusts another company to develop a better, more stable plugin than what Adobe itself produces so Adobe’s Flash plugin is still the ubiquitous one plugin.
    Publication of the Flash specifications signaled a clear direction for Adobe and their Flash product: they want to make money off the authoring tools, not the technology itself. This directional signal has been further reinforced by the diversification of Adobe Flash (less featured and lower cost versions) and Adobe’s strong efforts to extend Flash onto every available platform with variants including Linux, Macintosh, Windows, iPhone, Symbian, BlackBerry and Android with continued to attempts to get Flash onto ever more devices. In this context, the opening and broadening of Flash is the only course that makes sense. Any company can build Flash into their device, any programmer can build Flash extensions and any user can use and enjoy Flash.
    With the completion of the Flash foundational platform in 2003, Macromedia added the last vital features needed to make the broad base of developers and consumers happy with their product. In fact, every addition and development made by Macromedia and later Adobe, have been delivered in direct response to the requests, suggestions and demands of their consumers. It is, as Adobe is well aware of, in their best interest to continue to develop the platform to meet their customer’s needs. Any slip, any mishap, will allow room for a competitor to enter. Once a competitor has a toe hold, they will have a niche within which to establish and eventually erode Adobe’s position.
    While HTML and Flash may seem to be unrelated topics, the two stories, when compared, shed valuable insight on the processes and functionality of the two different styles of development: open (HTML) versus closed (Adobe), shared interest (or self-interest) versus vested interest (singular interest), responding to personal needs versus responding to customer needs.
    In the HTML development process, the WC3 had great success early in the development process. Even though the process has continued to remain open, the self-interest of the individual companies involved has driven the development process from one of harmony and continuity into a contentious frenzy with each company seeking to position themselves in the most profitable position possible. On the other hand, the closed (though transparent) development path chosen by Adobe has led to continued dramatic improvement over time, specifically responding to customer needs over time and focusing on building a solid business framework. There is no one but Adobe to be responsible for a good product (which will make them money) or a bad product (which will loss them money). In the closed environment, the needs of the customers, their actual needs as opposed to the perceived needs that would be represented and skewed in favor of the company demanding them, are the point of focus instead of whatever will make the company a core component while leaving others to rely upon them.

    (Thank you to Google and Wikipedia for providing the invaluable assistance in bringing all the details together for this essay.)