Total Pageviews

Friday, May 27, 2011

The inside story at Nevis Labs, Epilogue.

As I've described, there were some personal tensions at the lab. Nevertheless, I greatly enjoyed and felt privileged to be working on the project. My hours let me work alone most of the time, and I strived very hard to do things correctly. Everything considered, it was an awesome experience.

It was my intention to not say anything negative about any of the optics team, and I hope I didn't!

Todd and I had issues, but he was professional almost all the time, and he performed his job splendidly.

Tom also did a great job and he worked extremely hard. I think he was as essential to the success of the project as was Todd. Tom is not the type of person I would have spent any time with away from the job, but that is just a matter of personalities.

Bill disappeared, but that's his business.

I like Ken, and I'm sorry that our conflict at Christmas hurt our relationship. We remained cordial and have had some contact since I left NY.

People I really liked and who were always a pleasure to be with were Iliya, Nicola, and Regina, who cleaned spacers and had other duties, but whom I had not mentioned previously.

As I've probably already said, I am going to be very excited and proud when the NuSTAR satellite is launched, and I am very grateful to have been part of the team that built it.


The inside story at Nevis Labs, Conclusion.

Todd, the Assembly Engineer, had invented an adequate process for building the optics. It was labor intensive and held together with duct tape and baling wire in places, but it was effective. There was one critical assembly component, however, that was poorly designed - the strongbacks, which held the epoxied spacers on the glass until the epoxy had cured.

Each strongback was constructed of several components. The base was designed to apply even pressure along the length of the spacer. The parts that actually held the spacers were called shoes, and they are where the design was weak. The shoes were built from two or three pieces screwed together. There was a groove in the top for the spacer, under which was a Buna o-ring, at first, and then later replaced with Tygon tubing, a superior solution.

To hold a spacer in a shoe, a strip of Teflon tape was applied over the groove, and then the spacer was inserted. The problem with the shoes was that the width of the grooves was inconsistent, which resulted in some spacers being held inadequately. Sometimes the epoxied spacers would fall out when being attached to the glass, which resulted in a bit of a mess. The problem was limited to the double shoes, which were used where glass segments met. That was because the doubles were constructed of more parts.

Sometime in the fall Tom came up with a useful solution. If he discovered a double shoe in which the spacers were inadequately held, he marked it with a felt pen. When I saw that mark, I used two strips of Teflon tape on that shoe. It was a minority of the strongbacks that had the mark - two strips of tape on an unmarked shoe would result in the spacers being held too tightly.

Sometime after the first of the year there was a problem with the accuracy of the grinds. When Todd returned to Nevis, he embarked on a project to rebuild each shoe. That's when the o-rings were replaced with Tygon. Todd spent several very long days modifying the shoes.

I was happy about the modifications, because I thought I would no longer have to apply double tape to the marked shoes. I did pay attention, though, and if I noticed that Tom had added a layer of tape the previous day or if I thought a spacer was loose, I marked the strongback myself and subsequently applied two layers of tape. That sort of worked, except that in one or two cases where two layers of tape fixed one slot in a double shoe, it made the other slot too tight. Tom pointed those shoes out to me, and eventually only a few of the doubles were marked. None of the singles had been marked before, and none was marked now.

It was a few weeks before Tom mentioned to me that spacers in a few of the unmarked double shoes were loose. I asked him to mark them, as he had previously, but he said he was too busy to do it. (Both Tom and Iliya had an idle hour or more when the glass and spacers were being automatically scanned.) He wanted me to determine the shoes that needed double tape. I used the standard test, which was to shake the strongbacks upside down to see if a spacer fell out. I thought I had identified the loose shoes, but I hadn't gotten them all. Tom had a lot of experience in evaluating the shoes, but he had become annoyed and was not willing to help. Ken pushed Tom to mark the shoes as he had before, and Tom relented, but he did not mark any shoes. He was making one of his infamous stands.

Finally Tom and I got together to discuss what could be done. That's when he told me that the modifications that Todd had made didn't affect the width of the slots, so I should be double taping the same shoes as previously. It would have been good to know that a couple of weeks earlier, but no-one had told me, even though it was obvious to them that some of the marked shoes weren't double taped. After I told Tom I'd double tape all the marked shoes, he said, "The single shoes are okay. [pause] No, the tape on them has to be changed more frequently."

Then, mysteriously, spacers started falling out of single shoes, even though the grooves were the same width, the tape was the same, and no spacers had escaped from single shoes in the past (except for my mis-inserted 1.6 mm x 1.5 mm spacers).

The normal procedure was to replace the tape if it was torn or if it had been pulled loose when the spacer was removed. During my training, Iliya told me that he had changed tape on all of the shoes every week, regardless of condition. I was doing twice as many shoes as Iliya had ever done, but I worked out a system to change the tape on the spacers on each tray (there were six trays of strongbacks) on a specified day each week. Now Tom was complaining about loose spacers, so I modified my routine to change the tape a minimum of two times a week. Tom said that was inadequate. Since no spacers had fallen out previously, even with only a weekly change, I was aware that Tom was having another emotional reaction, and I wasn't sure how to handle it. I asked Tom to mark any single shoes in which a spacer was loose so I could double tape it, but he refused.

I asked Tom why the spacers had started falling out, and he decided that some of the spacers were curved, so that the ends were up too high in the shoes. Why that caused them to fall out, I don't know. More important is that I could see no evidence that any spacers at all were curved. Some had slight curves along the 1.2 mm axis, but they were uniformly straight along the 1.5 mm axis.

The evening of March 2 Todd sent me an email:

>> Kurt,
>>
>> We are still on L132 on FM1. We had a spacer not bond correctly because it partially fell out of a shoe and bonded crooked on the glass. There is only one strongback on FM1 and half the guidewheels. I've set up the grinds for rough and finish grind.
>>
>> If you have another hic-cup like we did this morning, just leave the machine as is so I can try and diagnose it in the state where the error happened. I think this mornings issue was with Windows and not Delta Tau but I can't be for sure. Also, don't change any of the grind parameters without first consulting Ken or I. Since I had to completely power down FM1 machine, all of the parameters that I've updated revert to the settings as built by Dan at layer 1. This means that the machine now thinks it is grinding layer 1 which would be disastrous.
>>
>> FM2 will advance to layer 126 and take two grinds as usual.
>>
>> Thanks,
>> Todd

I responded:

> Todd,
>
> Okay.
>
> I haven't changed the grind settings in months, except for the time a couple of days ago when I did a regrind. The last time I touched FM1 was when I was adjusting for the previous day, as you told me to. I'll stay away from them.
>
> I really don't know why spacers are falling out of single shoes all of a sudden, when they were fine for so long. I'm replacing more frequently than Iliya did. A couple of weeks ago I was in the lab with Tom and Ken and Tom was talking about the doubles, and he said, "The singles are fine. Well, no, you should change the tape every couple of days." I don't know what's going on, but I told Tom to mark any shoes he things are loose so I'll double tape them. He doesn't want to do that.
>
> Kurt

That seemed like a normal exchange.

The next morning I arrived at the lab and checked my email from the gowning room. That was my normal procedure, because Todd often sent email instructions in the evening after I'd gone to bed. Todd had responded to my reply with one sentence: "What time do you estimate you'll start the second grind on fm1?" The meaning of that was clear to me: Todd wanted to come to the lab to discuss the spacer issue.

It was March 3 and my last day at the lab was scheduled to be March 11. FM1 would be complete after one more layer, and I'd already been told that I could go home when only one optic was being built because the guys could easily handle what I'd been doing. I knew that it was inevitable that if Todd came in, he would side with Tom regardless of the circumstances.

After several minutes, I made a decision. I took my keys and left them on a table in the lab, then went home. Iliya, Tom, and Todd had all quit previously, and now it was my turn. The difference was, I had no intention to return.


Thursday, May 26, 2011

The inside story at Nevis Labs, Part 7.

When the school year began I started going into work earlier to avoid a traffic jam on the way home, and things quieted down in the lab. I was leaving about 20 minutes before Tom and Iliya arrived, but when I was delayed I was happy to see them. I didn't see Todd often, but he put aside whatever feelings he might have had and acted professionally. I did the same.

We were told about a month in advance that we'd have two weeks off at Christmas - the week before and the week after - mostly because of a shortage of glass. It would be a welcome respite from our 6-day-a-week grind.

On December 14 Ken sent an email that included:

**********
As most of you know, we are suffering problems with the schedule for building FM2. Iliya has offered to build during the Christmas week, so I am trying to figure out if we can support him. This would require Zeshan to mix and lay epoxy, Kurt to remove hardware, Doug to scan glass, and Marcela to stage glass.

To see if this is possible, let me know your availability on the following days that Iliya could possibly work. We would not work all of these days even if it were possible.

Sat Dec 18
Sun Dec 19
Mon Dec 20
Tue Dec 21
Wed Dec 22
Thu Dec 23
Sun Dec 26
**********

I wasn't excited about it, but I told Ken that I'd be available when I was needed. A few days later the holiday schedule was posted and I was working ALL of those days. I felt betrayed. I had volunteered based on the idea that we'd only be working a few of the days. It was particularly important to me because Twila was going back to Colorado on January 4, and I wanted to spend as much time with her as possible.

I assumed that the decision to work 7 days had been made upstairs somewhere. Since Ken was whom I felt most comfortable with, I wrote to him to say that I was irritated that I'd been misled. He told me that he was responsible for the schedule, and I was shocked. I reminded him that he'd said we would not work all of the days. I expressed some anger, then told him that I had vented and that was that, no hard feelings. But it did affect my relationship with Ken, who was considerably more distant thereafter.


Colorado wildlife.

Every spring about 22 gadzillion Miller moths fly overhead on their way to the mountains. Not surprisingly, about 21 gadzillion of them end up in our house. They don't really cause problems, other than flying into our faces at unexpected moments and cluttering up the floor. They do, however, fly back in the late summer to wherever they came from, and they like to drop in on that trip, too.

"Miller moths" is kind of non-threatening. "Army cutworms," OTOH (clickable)...



Wednesday, May 25, 2011

The inside story at Nevis Labs, Part 6.

I suppose I should mention something else that made my first couple of months uncomfortable - a UTI, urinary tract infection. For those of you who have never had one, here is some good advice - don't get one. Apparently I'd contracted mine sometime before leaving Colorado, possibly because of a long bike ride, and it caused a lot of discomfort on our drive to NY. The primary symptom was a sudden and almost-impossible-to-control urge to micturate. I'll omit the messy details, lest I be accused of yellow journalism, but a UTI is not recommended when working in a cleanroom in a bunny suit.

The UTI symptoms would go away for a week or two and then come back, and I kept hoping they would just go away for good, especially since the nearest Kaiser was in Washington, D.C. Finally my doctor talked me into going to the local Urgent Care Center, which gave me antibiotics. The medicine failed, so I went back to the UCC. Besides the $100 co-pay, Kaiser said they would only pay once for each condition. Fortunately, the people at the UCC were sympathetic and called the second visit something related, but sufficiently different. I saw a doctor who gave me different antibiotics and warned me that if I didn't get better that time, I would have to visit a urologist to see if I had a tumor in my bladder. Needless to say, I was much relieved when the second course of antibiotics was successful.

Shortly after the blowup with Todd, he changed my procedure to incorporate adding a calculated variable to the grind parameters. A couple of days after I started doing that, Todd came roaring in early and immediately inspected the machines. That made me uncomfortable. He asked me for the values I'd input, which, fortunately, I remembered. I couldn't escape the feeling that he was trying to trap me, since it would have been much easier for Tom and Iliya to enter the variables themselves. A couple of days later Todd did give that responsibility to the guys.

Going to the relatively isolated lab building at 3:30 AM had the potential for creepiness, especially if one of the outside lights were out. That didn't bother me, though, until I started wondering whether Todd had a firm grip on sanity. During our dinner with him, Todd had bragged to Twila and me about teaching his wife to handle a pistol. It's embarrassing now, but I became frightened. I wrote to Trish and told her that I was afraid of Todd. She asked if I wanted her to speak to Bill, but I declined to go that far. I just wanted it on record whom to question first if I ended up lateral in the vicinity of the lab. After expressing my fear I got over it.

A very nice woman named Nicola had some quality control responsibilities. One of her tasks was to enter the lab every day and expect the fillets, which was the distribution of the cured epoxy between the spacers and the glass. One day Tom came in and began loudly complaining that Nicola was slowing his work, and that he wanted her kept out of the lab. It turned into a big brouhaha. When I asked Iliya about it, he shook his head disgustedly. Faced with the reality that Nicola would continue to come into the lab, Tom threatened to quit. I don't know how that was resolved. A short time later Nicola found a problem with a fillet, which saved bigger problems later. I mentioned to someone (name withheld) that maybe now Tom would be happy that Nicola was doing her job. The person said that with Tom it was an emotional issue that probably had no practical solution. I laughed, not imagining that I would be in Nicola's shoes, so to speak, before long.

Tuesday, May 24, 2011

The inside story at Nevis Labs, Part 5.

A couple of weeks after I started working at Nevis, I met Ken, the Cognizant Engineer of the project. Eventually it was explained to me that NASA has a "Cognizant Engineer" on every project, and that was the best explanation I could get. Todd and Ken both live in California, and they rotated being onsite at Nevis, with Todd staying two weeks and Ken staying one.

I liked Ken right away. It is my impression that he is uncomfortable initially in social situations, but he can be the life of the party once he relaxes. He is smart and interesting. He was kind of the worrywart of the project, always afraid that something might go wrong. That was a little funny sometimes, but he made many improvements to ensure that nothing *would* go wrong. I always looked forward to his being there, and not just because it meant that Todd wouldn't be there.

Todd and I both tried to establish a professional relationship. I was still hoping that the initial flareup was an aberration. I bought two musical DVDs from Amazon that I planned to give him when the times were right and I was as friendly as I could be under the circumstances. Things seemed to be getting better, but that didn't last long.

We had started building FM1, the first flight module, about a week before. To add strength, the spacers for the first five layers are a bit chunkier - 1.6 mm x 1.5 mm (0.1 mm = 0.004"). The regular spacers are 1.5 mm x 1.2 mm. When picking up a regular spacer, it is rather easy to see which side is wider, but for the 1.6 mm x 1.5 mm spacers the procedure was to measure each one with calipers before inserting it in a strongback. That's what I did, but I soon realized that I could see which was the wide side by rotating it in my fingers. That amazed me, because my eyesight is hardly what it used to be, but I was wearing 2.0x reading glasses and the transition between the long and short sides was not especially difficult to see. I began to choose the long side visually, then check with the calipers. After I'd correctly oriented about 100 spacers without making a mistake, I announced that I could do it visually and didn't need to use the calipers. That didn't go over well, because no-one believed that I could do it, but I was allowed to continue without the calipers. Todd and Tom and Iliya all warned me that there had been problems in the past with visual alignment, but I thought it was cool that I, who has increasingly become Mr. Magoo with the passage of time, could do it.

That all blew up in a spectacular way. The morning of August 11 I checked my email when I got to the lab and found a message that Todd had sent the night before. He said: "Tom and Illiya had some problems while laying glass due to the spacers not installed properly. Two of them fell out of the shoes as Tom was loading them. They had already been epoxied and it makes quite a mess on the glass not to mention the loss in spacers and time to clean it up. After those two were discovered, they went and measured the orientation of the remaining spacers and found 5 more that were not properly oriented. Tom mentioned that he has to measure them to orient them properly where you didn't think you needed to. For 1.2 mm spacers, from this point forward use calipers and find the correct orientation during installation. Tom and Illiya will check the orientation before laying epoxy." [Notes: Todd always spelled Iliya's name with two L's, for some reason. Also, he probably meant 1.6 mm spacers, not 1.2 mm.]

I was willing to accept that I might have made one mistake, but I knew there was no way I'd made seven. I responded: "With all due respect, that's BS. It is easy for me to see which way the spacers go in, and I've double-checked 50 times and never been wrong. Nevertheless, I'll use the calipers... This would be a good time for you to find someone else for this job, if you don't think I can do it."

It had been one thing after another and I was really upset. When I finished work I went home and pondered what possibly could have gone wrong. Suddenly I had an "OOPS" moment. I sent an email to Todd asking, "1.6 mm is the height of the spacers, right?" He wrote back and said that the height of all the spacers is 1.5 mm, that 1.6 mm is the width. That was the only thing I'd been able to come up with, and, unfortunately, it was true: I'd inserted *all* of the 1.6 mm spacers incorrectly. As well as I can remember, no-one had ever actually told me which way they go in, and I'd made an incorrect assumption.

At that point I received a phone call saying that my father had died. I was pretty much a mess. Thinking that it would be hard for them to fire me on the day my dad died, I took advantage of the sad situation and zipped an email back to Todd angrily saying that obviously there had been a training problem.

That was probably the low point of my time at Nevis, but things didn't get a whole lot better.


Monday, May 23, 2011

We're here!

This is just a reminder that we're still here. I've finished about half of the next post in the Nevis Lab saga. Twila is still contemplating a theme for our next mutual blog. We've been hiking, which is my training for our planned assault on Pikes Peak in July (based on physical condition only, Twila could probably climb Everest in July). I've been spending more time with my helicopters, which is a lot of fun. There are still a lot of things to take care of, from pinball repair to truck maintenance to boat modifications and numerous household chores, as well as a rebellious toilet and the usual computer hassles. Life seems busier than ever, but, except for the Sharks choking like dogs, things are going fairly well.


Friday, May 20, 2011

Idea.

It doesn't matter how you feel. That always goes up and down. What matters is what you do.

Thursday, May 19, 2011

The inside story at Nevis Labs, Part 4.

Before I accepted the NuSTAR job, Carl warned me that pressure levels would rise as the project neared completion. Not only was the schedule tight, but also there was competition for funds between the different project groups. NuSTAR had started once before and then been abruptly canceled, with the project manager learning about the cancellation at a press conference. Government and warm fuzzy feelings really don't go together. There was tension also because of the precision required in the assembly of the optics. Any lapse in concentration could result in broken glass, contamination, or misaligned mirrors.

I didn't have to wait until the end of the project to see emotions arise. My very first day inside the lab, when I was at Nevis during my interview trip, Tom snapped briefly at Iliya over some minor problem. I really didn't think much about it. During my interview the next day, though, Bill told me that Iliya had quit. I was flabbergasted. Quit? Iliya came back, of course, but during my time in NY everyone who worked in the optics lab (except Ken - he might have quit, but I didn't hear about it and he doesn't seem like the type) quit at one time or another but then came right back. Not me, of course. When I quit, I'm gone.

I was responsible for an entirely new level of tension in the lab. There were two components to that. One was that at first no-one was really happy that I was there. They'd made the decision that they needed someone to do my job, but naturally they weren't excited about someone new being in intimate contact with $4M worth of extremely fragile optics. The other component was my personality, however one might view it.

The first disaster occurred when I emailed Todd my timecard on July 24, just 3 weeks into the job. I added a note saying I felt he didn't trust me because of my background as a software engineer, then proceeded to outline my non-software experience. I ended the note with: "Anyway, I do know how to do mechanical things, I'm not a cowboy, and I'll only do what I'm told to do. So relax a little and have some fun!"

Well, that didn't go over very well. Todd (who was in California at the time) wrote back:

"hmmm.....What concerned me is that you said you could run the routine of dressing the wheel and finding it's position. These machines are very unforgiving. I don't doubt that you could find the right program to run but I don't know if you know how to touch off a grinding wheel with the dressing tool, dress the wheel with the diamond tool, null the lvdt, knowing what speed and settings we use to gentle position the spindle, touch off on sacrificial which is a very friable material and know that you are not grinding it while touching off, all the while there is a 2M$ optics sitting there that we are trying hold micron position from layer to layer. The absolute most critical step is finding the grinding radius of the wheel. No one should feel comfortable doing that and that goes for me as well. It Tom more than a month to get used to the controls on the pendant and he is still un-nerved. Ken will not even attempt to dress or find the wheel even though he has seen it dozens of times.

So it bothers me that you think you can do that when I've haven't let you, much less, trained you to do those tasks. Believe me, it is not as easy as it looks and it surprised me that you thought you could do it.

I hope I've explained myself clearly and when these optics are in the hands of the spacecraft contractor, I will relax. Until then, I take this production of the optics extremely seriously and I expect the same from those working for me."

Both Todd and Tom had both given me lessons in dressing and finding the wheel, which somehow had given me the idea that I was expected to do it at some time. Apparently that was not a universal view. It really isn't a complicated job, but I had taken copious notes and paid close attention. I had no doubt then and I have no doubt now that I could have done it as well as anyone.

Todd and I had established a hostile relationship, and there proved to be no way to fully recover.