How we did Christmas for the City Streaming (The Technical Details)

For the past 2 years, my church has hosted an event called Christmas for the City. A few hundred volunteers and about 30 other churches partnered with the event from around the area to take over a downtown building for a few days, providing support and resources for people in hard times in Winston-Salem, and a really unique Christmas experience composed of music, art, dance, literary works, and a ton of other features.

One of the goals for 2009 was to bring the event online in a bigger way with a few online streams that would let a virtual visitor surf through some of the main areas with just a couple of clicks. This blog post describes the process of how we ended up with what we did, and some of the challenges to solve for the next event of this type.

The Stream Team

This year’s streaming was very much a team effort, and everyone involved donated a lot of time and resources into making it happen. It wouldn’t be right in finishing this post without props going to:

  • Sean E. – Sean serves as an engineer for Microsoft and provided vision, hardware, some Windows 7 boot-to-VHD ninja, and way too much effort working with Time Warner Cable.
  • Sammy E. / Triad Tech – Sammy, with Triad Technologies, provided wiring through 3 floors of the Millennium Center, giving us the backbone we needed to establish a real network.
  • Jason H. – Silverlight kung-fu, getting the initial application prototype built and into the wild based on some of my very “loose” specs…
  • Brandon M. – Streaming support throughout the event
  • Dillon M. – Mobile Cam Driver
  • WSF Media Team (Aaron R., Chris E.) – These guys rocked the tech world with some awesome animation, and provided cameras and various other accessories and support.
  • X9 Technologies – X9 provided the network infrastructure, access points, servers, bandwidth, and code to get video out of the building and across the ‘net.

Location, and Wireless Networking

MillenniumCenter-NuclearFalloutShelter The Millennium Center in downtown Winston-Salem is the location of the event. It’s an extremely “historical” building with lots of “character”. We needed a solid wireless network to build on, and it didn’t exist in this building. The “character” (marble floors, walls, solid ceilings, 3-ft thick plaster walls) meant 2 things – first, it was designated as a nuclear fallout shelter, and second, RF doesn’t work like you would expect in there.

 

CiscoAironetSince wiring is pretty difficult and we only really  had one point of access to each floor, the strategy we went with was to cover the “main” area of each floor with one Wireless Access Point. We used three Cisco Aironet AP’s. Power was really not available or not convenient next to the network drops, so we solved this with the magic of Power over Ethernet (PoE). To get the best performance out of these for general internet access (below 5mbit), they were optimized for range instead of speed.

Windows Media vs. Flash

Of the streaming options out there right now, the two most viable options are Windows Media and Flash. When it came time to decide between the two, I was kind of 50/50 on both, with a slight tendency to go toward Flash, just because it had the larger install base and generally “just works” (most people have watched a YouTube video on their computer, so they already have what they need). Sean suggested using Windows Media with Silverlight. While we probably could have accomplished the same thing with Flash, we decided to create the player interface with Silverlight, and be able to pretty carefully control the user’s experience. The decision to go Silverlight over Flash pretty much came down to Silverlight and the underlying .NET stack offering us very rapid development that we could rely on without a whole lot of testing and tweaking. Although we knew some users would need to go through the Silverlight install, it was worth the tradeoff.

The Interface

Here’s a screenshot of what the final looked like:

CFTC Streaming Screenshot

The interface is pretty straightforward. The user can select between 3 different camera streams (originally Party Room, Performing Arts Stage, and Multi-Cultural Room). In the upper right hand corner, a real-time scrolling schedule is kept in sync with the event as it happens, so the user can check out whichever scheduled artist they want. In the lower right hand corner is the Twitter integration, displaying tweets from Twitter’s search API that contain the hashtag #CFTC.

Testing

I had a little bit of dumb luck when it came to testing this application before the event. The Friday before, we had a pretty good snowstorm here in North Carolina (which never happens), so I used that opportunity to set up a camera and turn it into a “snow cam”. This gave people a reason to go and visit the application to see the snowfall, and gave me some valuable feedback via comments and logs of how many people had Silverlight installed, and whether or not everything was working as expected. Surprisingly, not one person had a problem viewing it out of a couple dozen. Success.

Failover Streams

The only problem with Windows Media Services is that when a stream ends (i.e. in the event of a network interruption), it gets pretty ugly. The player usually just freezes on the last frame, or displays an ugly error message, which is obviously not ideal. The first night, I ran out of time to implement a static failover stream, but by the second night I was able to have it in place.

This is a fairly undocumented method in Windows Media Services, but it requires 2 publishing points. First, the publishing points for the “live cams” (we called them CFTCCam1, CFTCCam2, and CFTCCam3), and second, the publishing points for the “broadcasts” with failover (we called them CFTCBroadcast1, CFTCBroadcast2, and CFTCBroadcast3). The “broadcast” publishing points are actually a Windows Media Server Side Playlist (.WSX file) that contains a <switch> statement with a loop. So, the Broadcast publishing point will try to play the “live cam” feed first, but if it’s offline, it will switch over to a static .WMV animation file (the Christmas for the City animation).

The only caveat by doing this is that you really have to keep the “failover” file short, as it must play through all the way until the server attempts to switch back over to the “live” stream. We kept ours down to about 25 seconds (the animation followed by a quick “hang tight!” message), and I considered this acceptable.

Encoding

ExpressionEncoderEncoding was done with Microsoft Expression Encoder 3. This is a fairly new product in Microsoft’s Expression line of design tools, and it’s a fairly slick interface (major improvement over Windows Media Encoder 9).

We had 3 computers (2 desktops and 1 laptop, 2 running Windows 7 and 1 running Vista), with Sony HVR-1 cameras using FireWire for input. This gave us a pretty good video source with the camera audio taking XLR output from the audio console(s) or camera mic.

I have to say, while Expression Encoder is a great interface, it had its share of “isms”. First, after installing it on a clean Windows 7 or Vista machine, using it in Live Encoding mode with our FireWire cameras, it would crash hard every time we pressed “Start Streaming”. This was fixed pretty easily by installing a QFE found here. Also, it generally does not deal with network hiccups very well. Again, we’re 100% wireless in a nuclear fallout shelter, so whenever there was a hiccup, the stream would die, and our only clue would be a little line of red text next to the Publishing Point output with “Unspecified Error”. Also, the “remember password” option when setting up Publishing Points just plain doesn’t appear to be hooked up, so this became an annoyance with our volunteer team that helped set up and maintain the streams throughout the nights. Overall, these were just slightly annoying problems that I’m sure will be fixed in time in an otherwise great piece of software.

Stats

We watched stats in realtime during the event through X9’s Streaming Client Center. This hooks into the backend of the streaming servers and cross-references IP addresses with X9’s Geomap database to give us an idea of our traffic on each of the 3 feeds. We could also overlay the data on top of a Google map to impress people.

 imageimage

The Mobile Random Cam

Watching our stats on Monday, while people were frequenting the Party Room and Performing Arts Stage, we noticed that no one was really watching the “Multi-Cultural Room Cam”, which is a camera in a room with an 8-ft ceiling and is pretty narrow. There was no board audio feed on it, and most of the time it was just people standing around in front of the camera.

Tuesday morning, I was helping unload 25 bags of Chex Mix and 20 gallons of Apple Cider from my wife’s car into the Millennium Center, and was attempting to use a caster plate the production company used to cart around truss. By about the 3rd time everything had fallen off, I got frustrated, and went and found a large catering cart. Pushing the cart around, through the elevator, and on to the second floor, I was thinking “man, wouldn’t it be awesome if people online could just virtually ride around on a cart like this……”, and later that evening… ta-dah:

RandomCamCart

On the back of the cart is a laptop running Expression Encoder, and on the front is the Sony cam. Each laptop would last about 2 hours, so we switched between Sean’s small Lenovo and my personal Dell Precision. We needed something to get the camera lens above the cart handle, so I found a cardboard box. Dillon found a bungee cord in his truck to keep the camera from flying off. The Mobile Random Cam was born.

The stats on the mobile cam were pretty funny. Viewers immediately loved it and started watching it more than the other 2 static feeds. Then we’d go into an elevator, lose our WiFi signal, lose all of our “fans” to the other feeds, and they’d join us back a few minutes later. I think the random-ness combined with the comedy of watching as Dillon and I try to navigate this cart through thousands of people, up and down stairs, and interviewing pretty much everyone we ran into made this stream pretty entertaining. Oh, and we had a blast doing it.

That’s It.

This pretty much sums up the CFTC streaming… by all means, it was a great success, and our team was awesome. We definitely pushed the limits of wireless, Expression Encoder, and put a few miles on a catering cart. Can’t wait to do it again for another event!

  • Only got to watch a little bit but you guys did great! I was quite amazed at the random mobile cam and how well it all worked. Very cool indeed!

  • Colleen Wenner

    Great to hear the details - some of which I used to be a part of - the chex mix anyway!

    You did a fabulous job and I have to say that every church needs someone with your attitude to just figure things out no matter what it takes. I'm so sick of the "not sure we can" answer!

    Love ya guy! Give that wife a big hug and tell her to relax! It's over!!!!

  • Pingback: ()