Before the Dawn of IP Telephony - Part 16Frozen by discovery of fatal problem (latter half of 1999)
These contents translated a serialization article carried by ITPro IP telephony ONLINE published by Nikkei Business Publications, Inc. Jump to the original (Japanese).
eSound Venture Unit
Oki Electric Industry Co., Ltd
After the transfer to the affiliated company, I became a leader of the team developing component parts for OKI's IP-PBX called Line Unit (LU). Then, I discovered that I had taken charge of a task that no one dared.
Photo 1 : LSI exclusively for VoIP
Developed by OKI
In June 1999, one month after I was transferred to the affiliated company, an evaluation sample of LSI for processing VoIP was delivered. It was a single chip exclusively for VoIP, the first achievement of its kind by OKI (photo 1). The chip was built with features necessary for IP telephony and gateway, and the development scale was particularly large for an LSI. We started evaluation by mounting the LSI sample on a board.
This LSI was incorporated with the concept of "pursuing the best voice quality under the installed environment" we announced at ISS '97 held in Toronto, Canada in 1997 (See article in previous series). The LSI was built with a technology for establishing voice quality over LAN and it was the key component of our IP-PBX line unit (LU). It also meant that the LSI was so crucial that the entire development project may be aborted if a fatal bug was found.
There was the need to proceed with the evaluation with caution. On the other hand, there was no time to waste. Although this is a dilemma that always occurs during product development, there is never a sufficient margin for development when there is a shipment deadline for the product. When entering early July, the time for mass product judgment of the LSI was coming near, while the evaluation was not complete.
Due to the bitter experience of discovering a fatal bug in the core LSI during the development of VOICEHUB in 1995, I obviously felt strong about performing a thorough evaluation. In order to achieve the goal of starting shipment of IP-PBX at the year-end, however, there was the need for mass production judgment of the LSI in July. Although the LSI had minor issues that could have been improved, I saw no reason for objecting to mass production since no fatal problems were found. I agreed to the "go sign" and the LSI entered the mass production stage.
Honestly speaking, I wasn't aware that the decision was so important at the time since I joined the project during the middle of development. Furthermore, the development of IP-PBX was widely known even at OKI. Different from the development of BS1100-VOICEHUB, the first VoIP product, there were expectations to release the unit as a new VoIP product group of OKI. There was no way that the development could be delayed.
LU development continued. In July, a series of verifications for the actual hardware was completed and everything seemed to go smoothly. Although a few problems were found during the verifications, they were mostly parameter setting errors of the LSI or peripheral circuit errors; nothing that pointed to the design of the LU itself. These are errors that can be solved with sufficient verification.
I as well as persons involved in device designs take on the position of "suspect your own design before suspecting a bug in the LSI." Development members centering on Koji had a thorough grasp on the events and contacted the team in charge of the suspicious area only when there is sufficient evidence. Completion of a product is the result of finishing the area in charge to perfection. Hence, the method is correct.
All verifications were near completion. There was one event, however, that remained unsolved. We thought mildly about it, assuming that it was caused by the small operational margin of the design circuit like the other errors. But there was no sign that the problem would ever be solved. I started to feel sick in my stomach. And once again, I heard the report, "It's an LSI bug." There was no mistake about it, since the report was made by Koji, who never fails to get the hard facts, and nails down all other possibilities.
Although I cannot explain the problem in detail, it was caused by a technical method used to pass signals in a line exchange network that is normally synchronous, however in an IP network it is normally asynchronous. For communication among offices, there was a concept that all communication should be guaranteed for signals other than voice regardless of whether it was a fax or modem. This was an important technology for synchronizing signals within an asynchronous network. And solving this problem was indispensable for the communication of various media signals without intermittence.
Passing a thread through the eye of a needle
The LSI was already in mass production, and modification was out of the question. Creating the LSI from scratch would surely mean delay in product shipment as well as further development cost. Although measures were discussed with the development team of the LSI, the essence of the problem was deep. The measure established through one month of deliberation was extremely delicate.
When shipping a product, the operation must be absolutely guaranteed. The measure was to control signals within an extremely small range. The idea was like constructing a slender building on a narrow strip of land, or passing a thread through the eye of a needle. However, creating this signal would mean absolute guarantee of operation. The LSI bug of VOICEHUB back in 1995 required the creation of an entire circuit on the outside to substitute for the faulty circuit-much like an organ transplant-since such measures as the above could not be taken. In comparison, the bug we had at hand was severe, but not a fatal one.
There are no LSIs without bugs. As a device designer, it is crucial that we learn to deal with such LSI errors. Although we suffered to the end from this problem, the schedule for commercialization was set without drastic shipping adjustments. With the worst part seeming over, I was just feeling a sense of unity with the other members of the LU development team. I also had a sense of mission since the product had to be released and that I was at one point told by the person responsible for product planning that I would be fired if I couldn't do it. But by this time, I knew I could do it.
Frozen by the fact
When entering September, preparations for verifications of the actual device were being made by the software development team as well. This was because packet processing and signal processing of the part handling voice were realized by software. In October, creation of various procedure guides for design verification, or test design, had started. But we were stricken with an unbelievable fact immediately after entering this test design. There was a great possibility that the absolute requirement of "establishing voice quality" would not pass. When I was notified this, I felt my spine suddenly freeze.
There may be a problem establishing sufficient voice quality. When I heard this, I thought that there must be some mistake. I questioned myself repeatedly. To my disbelief, there was a condition that we didn't take into account. Engineers familiar with line exchange networks would have noticed it relatively easily. Or maybe, I was thinking too much. I decided to look over the references of LU method examination. The voice processing software was in the middle of development. There might be a chance of recovery.
I was disappointed to find no traces examining the particular condition. It was a fact that I immediately wanted to turn from. The problem was related to processing capability. And the only option for solving this problem to my knowledge was to create the LSI again from scratch. We might just screw up on this one...
But there was no point of grieving over a fact. I decided to look at the results of sound verification-for the faint glimmer of hope.
Issues of the fatal injury
Photo 2 : Anechoic room, the low building on the right
In December, evaluation of the software-embedded system was started. This is the point where the voice quality would be assessed at last. And as I had suspected, my fears turned to reality. Sufficient performance could not be accomplished under certain conditions. Tests were repeated over and over with persons in charge of developing the LSI in the anechoic room (photo 2). But it didn't work. No measures could be found. We also called in-house members familiar with analog circuits and the big-guns of communication for their insight. But it still didn't work. I wondered if there was an error in the method design.
We may not be able to make it for shipment. The unmistakable image of failure crossed my mind. I seriously thought that the former manager who dared not touch this project had been right all along. There was no solution no matter how we tried. We were hopeless. But a savior had appeared from where we least expected. And I was stricken by the true potential of engineers who had a passion for pursuing "sound."
... To be continued