EPISODE 94

Intercontinental Exchange is Building Tomorrow’s Trading Platforms Today

67 minutes · April 01, 2019

What does it take to keep the engines of capitalism running, while simultaneously developing state of the art technology to transform global markets? Diving into this racy topic are Mayur Kapani, Chief Technology Officer at Intercontinental Exchange, along with Sridhar Masam and Alexei Lebedev, Head of Technology and Chief Software Architect, respectively, at the New York Stock Exchange. Find out how their career paths brought them together to build, operate, and oversee the technology that allows an exchange to process more than 100,000 messages in the blink of an eye.

Speaker 1:

From the library of the New York Stock Exchange at the corner of Wall and Broad Streets in New York city, you're inside the ICE House, our podcast from Intercontinental Exchange on markets, leadership and vision and global business. The dream drivers that have made the NYSE an indispensable institution of global growth for over for 225 years. Each week, we feature stories of those who hatch plans, create jobs, and harness the engine of capitalism. Right here, right now at the NYSE and at ICE's 12 exchanges and six clearing houses around the world. Now welcome, inside the ICE House, here's your host, Josh King of Intercontinental Exchange.

Josh King:

A few weeks ago, here in the library of the NYSE, I was talking to Denny Hamlin, the 61st winner of the Daytona 500, about what it takes to win the great American race. Of the 43 cars running, their engines consumed a collective 5,375 gallons of racing fuel from start to finish. But what does it take to keep the engines of global capitalism running around the world every workday? Our stock derivatives and futures exchanges are together one of the planet's most complex networks of machines and pure computer power making the winning car at Daytona look like a tortoise compared with the speed at which the exchange must operate flawlessly. In a global marketplace exchange groups bear the responsibility to be high performing and fully operational, regardless of market conditions, weather conditions or any unforeseen business impact you might imagine. Intercontinental Exchange earned its stripes for being at the forefront of the technological innovation.

Josh King:

At the New York Stock Exchange, it's managed to not only stand the test of time, but to evolve along with it. In a world where activity is measured in microseconds which to be clear is 1000000th of a second, efficiency, high performance and most importantly, resiliency are non-negotiable table stakes to compete in a global marketplace. Judging by the faces of the people here and the uniforms they wear, the NYSE is certainly a different place than it once was. In the elevators, you are just as likely to bump into a mathematics PhD or computer engineer as a trader. That's why today's discussion focuses on the technology behind ICE's 12 exchanges, six clearing houses, and the company's rapidly growing data services business called appropriately enough ICE Data Services. It's a wide ranging topic, which is why we're making room in the ICE House for not one but three special guests. Mayur Kapani, Chief Technology Officer at ICE, Sridhar Masam, Head of Technology at the New York Stock Exchange and Alexei Lebedev Chief Software Architect at the NYSE.

Josh King:

Their career arcs have all led Mayur, Sridhar and Alexei to their responsibilities, designing the state-of-the-art technology that has transformed global markets. Each has played a role in helping Intercontinental Exchange position itself as a leader in financial technology. CTOs and tech geeks out there, strap on your seat belts and start your engines. We're going out for a fast ride on the technology frontier, building the exchange of the 21st century. Our conversation with Mayur, Sridhar and Alexei, right after this.

Speaker 3:

Tencent Music Entertainment Group is the largest online music entertainment group in China. We operate the four largest online music apps including the social entertainment part. We are serving more than 800 million users. The user experience on TME platform is extremely great. I look forward to have a long term partnership with New York Stock Exchange. Tencent Music Entertainment Group is now listed on New York Stock Exchange.

Josh King:

When it comes to building, operating and overseeing the technology that allows an exchange to process more than 100,000 messages in the blink of an eye, where do you begin? Sitting across from me are the engineers tasked with making that magic happen? The three of them just blink their eyes, three and a third times, that's 1,000,000 more messages in the lifespan of an exchange. Mayur Kapani joined ICE as Vice President of Trading Technology in 2006 from the Philadelphia Stock Exchange and was named Chief Technology Officer in 2016. Sridhar Masam joined the NYSE in 2002 from the Pacific Stock Exchange, which is now a part of NYSE Arca. He was instrumental in building the exchange's current tech platform and was named head of NYSE technology in June, 2018. Alexei Lebedev, Chief Software Architect joined the NYSE in 2013 from Algo Engineering, the company, which he co-founded. ICE bought Algo Engineering in 2014, to drive the NYSE stock and options market.

Josh King:

With so much of that effort substantially and successfully implemented, a technologist like Alexei naturally looks to new adventures and computer coding and software development, which is why this conversation for him serves as both a retrospective on the work achieved, a meditation on some of the ahead, and a swan song from the NYSE as he prepares with our deep gratitude to follow new pursuits a few months from now. Sridhar and Alexei only had to travel down a couple flights of stairs to get here, but Mayur, you're in town from the ICE headquarters in Atlanta. As I mentioned in the introduction, you've all come from different walks of life, have taken very different educational paths and are even from different regions of the world, but you've come together through your Fintech expertise. Let's start there. How did each of you wind your way into a career in technology, Mayur?

Mayur Kapani:

To start, I actually started working for a retail banking software platform where I was tasked with programming ATM machines.

Josh King:

In the U.S?

Mayur Kapani:

In the Far East, Malaysia, Indonesia, China. I was working for Citicorp Software, a division of Citibank. From there, I landed in the U.S based on my skills of programming the ATM machines, where in a Philadelphia Stock Exchange, at that time, I was told that I had to take all these electronic orders and route them to different printers on the floor. That's how I got started there. From there, that system evolved to be a full electronic system. ISC had come about then where they were trading options electronically. Philly, that was one of the smaller exchanges at that time, to save itself from extinction decided that they needed to turn fully electronic. I was part of a journey where we moved from a electronic floor based system to a more hybrid and an electronic system that traded options and competed with the likes of ISC and CBOE.

Josh King:

Was technology always in the Kapani family business?

Mayur Kapani:

No. I come from a family background in accounting and finance. The word engineering was not well understood. It just so happened that I was one of the few guys who decided to study, got interested and went to a technical school, Indian Institute of Technology, Kharagpur and studied mechanical engineering. That's how I started my journey in the technology area.

Josh King:

Did you make your parents proud?

Mayur Kapani:

I think so. I think it was interesting. My dad, initially, when I decided to go and do engineering, he said, "Do you really want to do engineering? You're not going to make a lot of money." He was a tax and a finance consultant. He was always telling me that, you know what you go where the money is and not where you have to first learn a lot of things to make money. I kind of decided to go the other way. I'm glad I did because I really enjoy what I do and it's a lot of fun.

Josh King:

On that move from the Philadelphia Stock Exchange, was trying to get to the United States a definite goal, or could you have applied your engineering trade anywhere in the planet and have been happy?

Mayur Kapani:

Moving to United States was simply because that's where I got the opportunity. It was just before year 2000. There was a huge demand in the U.S for engineers. It just so happened that the technology I was working with a fault or run platform at that time was in high in demand. Philadelphia Stock Exchange was one that needed that engineering and I landed up there.

Josh King:

Sridhar, how did your path differ?

Sridhar Masam:

I started in 1997 right after graduating college at a company called Cognizant Technology Solutions. It's still listed company today in China and South India. They recruited me off campus. My first project was to work for the Pacific Stock Exchange. They were a client of cognizant at the time. I moved here in 1998. I worked there for a couple of years as a consultant to the company. Then I tried four different startups during the.com era being in the bay area at the time. Once the bubble burst, I went back to exchange where we did the similar to other options exchange at the time as a hybrid platform, we platform the Cobol mainframe based system into a Java based platform that I'm running on Windows server using SQL Server database, to be this acquisitions NYSE Arca, NYSE:ICE, I'm here in New York now.

Josh King:

Was that work for Cognizant at the Pacific Stock Exchange of first time in San Francisco or you've done other businesses?

Sridhar Masam:

No, that was my first time. I just worked out of college eight months in India. That was my first assignment here in U.S.

Josh King:

Did Silicon Valley blow your mind or you take it all on stride?

Sridhar Masam:

It did. That was one of the things where I... The Cobol at the time was more of an extinct type of language and I kind of retooled myself with Java. That's where I went to work for different startups with the web technologies at the time and then came back to the exchange built at Java based trading platform.

Josh King:

There's perhaps no greater time to be a computer scientist and to be exploring and pioneering the capabilities of machine learning. Alexei, as your time at the NYSE nears its conclusion after so much good work here behind you, what areas of coding and continued advancement in this space peak your interest? Any thoughts on where you'll lend your focus next?

Alexei Lebedev:

Well, that's a great question. It's one that I've been asking myself and I will probably have to continue to ask myself after I go. I don't have any specific plans at the moment. I'm basically trying to keep the field open, but I will almost certainly go back to working on the compiler technologies. For ALGO M2, I wrote a compiler that helped manufacture in a sense the code, because the secret to low latency is to simply construct the code without any glue that comes from using third party libraries. To do that in the short amount of time that you typically have to build all the features that you need, you just need a special tool. Such tools exist, various stages of software construction. There are compilers. The compiler that in a way, powers pillar and generates 95% of all the code that executes in production is an example of such a tool.

Alexei Lebedev:

I've been asking myself, for instance, could you write a compiler that would build an entire distributed application from just a single source file so that you would write something and it would automatically scale to all these machines and you wouldn't have to code that by hand. I don't know the answer to that. Maybe it would take a couple years to explore that area, but I think it might be very interesting essentially come up with a transactional application as a demo app.

Josh King:

Millions of people watch the opening bell of the NYSE and CNBC every morning. Millions more watch the crawl of data on the lower third of their screens quoted prices for ETFs, Brent crude, gold, coffee, and thousands of other commodities, futures and derivatives. There's an engine that keeps all of its straight. This computer aided thinking. Mayur, tell us what's happening behind the curtain, all of you.

Mayur Kapani:

Behind the curtain. Okay. In terms of what we do every day. Well, every day is paranoia. What is going to break. Every day is figuring out, okay, we're going to put new things in that is going to help provide a new feature or a new functionality to the customer. Have we done it right? Have we done it in such a way that it'll be useful? What will customers feel about it when they see it? A lot of anxiety, a lot of paranoia, and then the wait for the thrill when it goes live. First thrill is nothing broke. The second thrill is, okay. Now we have created something useful that is going to make a big difference to the markets and our customer base. Behind the curtain, lot of that happens. Also, a lot of work goes into planning, thinking and figuring out what are the next areas or what in incremental or non incremental, disruptive change we are going to try to attack next test technologists that's going to make a big difference to the way we do business.

Josh King:

Andy Grove had it right with only the paranoid survive if you're Mayur Kapani?

Mayur Kapani:

Yeah. It's a little more than that. Not only the paranoid survive. In our case, your job survives only when it stays up. For all of us, we take great pride in running super mission critical platforms. If we can't keep that running every day, all day, then we are not fit for this job. We really have, I think, a unique personality or you can say malfunction in our thinking to be able to do this job and sleep well.

Josh King:

Sridhar, Alexei, give me some scale of this that Mayur just described that you're paranoid throughout the day, but what is actually happening that you're watching this stew continue to reverberate from the beginning of trading until the close?

Alexei Lebedev:

Sure. Maybe for the listeners, I should describe what an exchange looks like just in terms of its sort of scale. Maybe 35 miles Northwest of Manhattan is an enormous data center with several floors of machines. In one hall of this data center, 100 servers occupying maybe the space of two or three refrigerators with 2000 processors inside are processing, roughly 3 billion instructions each second, each. That's several trillion instructions per second. Connected to this computing core are several hundred cables over which bites traverse back and forth representing orders and acknowledgements and trades. The job of a creator of an exchange is basically to specify to each of these thousands of processors, what to do at any moment in time. Once you've specified that you walk away and you're not allowed to touch it. If it goes down in flames, you're just an observer and you're going to regret it. You have to be pretty sure upfront that nothing unexpected can happen. There's a special art to that.

Mayur Kapani:

Just before it goes, anything goes out of flames. Alexei makes sure that we already have double and triple backup that takes up automatically just to be clear.

Alexei Lebedev:

Well, that's part of that art.

Sridhar Masam:

Between our equity and options markets on a given day, a peak volume we do around 75 billion messages. We're processing around 1000000 messages a second within tens of microseconds off latencies in order entry to customer acknowledgement. Again, just to add to what Mayur said, it's every day, every weekend, and every day the more opening and closing is crucial for us in terms of smooth opening of the markets, smooth closing, and make sure that we do a lot of effort on weekends to make sure the Monday, any new releases, installations we come up clean on Monday.

Josh King:

These volumes and this scale is not declining. It is ever growing. Mayur, you joined ICE in 2006, at which point it was still a relatively unknown Atlanta based energy exchange that was starting to make some marque acquisitions of historic trading venues, such as the International Petroleum Exchange and the [inaudible 00:17:09]. Little did you know the biggest challenge was yet to come. But start by telling us how you developed your acquisition muscle through those earlier deals and then we'll get to the big buy of the big board and the deal that brought the NYSE within the ICE fold in 2013. But there you are in 2006, and you're now going to go on a binge of acquisitions. How do you, as a technologist build the muscle to be able to do that?

Mayur Kapani:

In hindsight, 2006, I had no clue in terms of what I was getting into when I joined the ICE. I was part of an options exchange. At that time, ICE was a small company in Atlanta that was doing OTC trading of power and natural gas and we had just got into the IPE where we were trying to convert it to full electronic platform from a manual floor based platform. We started seeing the volumes. One of the the remits I had going in was let us leverage all the learning and everything that has been done by, especially the U.S equity exchanges and equity option exchanges in terms of building scalable infrastructures and apply this to the derivative trading.

Mayur Kapani:

Going forward, we started to look more and more like exchanges and Jeff's playbook was we wanted to convert all the manual processes that were associated with trading, where there were floors and chats and other ways, and means by which buyers and sellers got to got together into more electronic and more efficient form of using the internet and the accessibility that was available through it, to get a lot more people into this. Also, ability to distribute it much more globally, make them more global markets.

Mayur Kapani:

IPE, New York Board of Trade, we used our core muscle in technology to scale and make those markets global. We also, at that point started building a clearinghouse by starting with the clearing platform that we acquired as part of New York Board of Trade and launched the first UK clearing house in a hundred years. We also acquired at that point a chat platform that was an early on chat platform that had a network of traders who used to chat, and trade gas options and build out that platform and connected it and made it more electronic. All the workflows and processes that were related to it, made it much more electronic and digitized it such that we could bring in a lot more efficiencies into the trading of those instruments.

Josh King:

Sridhar, what was the opportunity that brought you across the country from the Pacific Stock Exchange to right here, the corner of Wall and Broad?

Sridhar Masam:

In 2014 and mid of 2014, Pillar was at the time was project Aby was the one which brought me to New York. I was asked to move here to lead the Pillar initiative. Most of my career, I have been in a building technology platforms, high capacity, low latency, high throughput systems, primarily exchange space. With each one of those three platforms, we went through many challenges and learned a lot of lessons. One of the things which we did with pillars to apply all those lessons learned and try to build a platform of the future, addressing all the concerns, what issues we had in the past.

Josh King:

Sridhar, the status of the Pillar project at this point.

Sridhar Masam:

We have NYSE Arca, NYSE American, NYSE National, NYSE Tapes B and C markets rolled out onto pillar. This year, we are targeting NYSE Tape A as well as CHX our new exchange onto the Pillar platform. We continue to build out on the Pillar options this year, along with Pillar SIP, which is replacing our SIP with the Pillar technology.

Josh King:

What's the SIP?

Sridhar Masam:

It's a Security Information Processor. This is an entity owned by multiple exchanges. It's a committee of its own, which is taking information or top of the book report from all exchanges and aggregates and be that center point where everybody can see everybody's prizes. Every exchange of prizes meet intermediate aggregation point for all the exchanges for every stock.

Mayur Kapani:

Yeah, and I think this is a good example, Josh, where we have taken the core technology, which has worked really successfully for us in a particular area, and we are using and leveraging it much more extensively in our other platforms and going as Sridhar mentioned, we are re platforming the whole SIP, which is now based on 10, 15 year old thinking on architecture to a much more modern, much more efficient and compact low latency platform, which we have built over the past few years.

Josh King:

At what point, for any of you, do you declare the Pillar Project fully completed or is it constantly reinventing itself and finding new applications to build onto it or reducing the latency even further?

Sridhar Masam:

I would say it's both. It's never done. We are finding our market data systems. We're trying to bring it on onto Pillar. We already have our audit entry gateways, along with our matching engines onto this platform. It's ever evolving and we have continuous R&D looking at innovative technologies, how with that can be augmented with Pillar, be it new hardware, be it engine architecture and design, constantly evolve the platform and try to make it more of a platform that can be leveraged across multiple applications. We're trying to get into parts of Pillar to be more of an open source where people can start contributing and using it.

Mayur Kapani:

Well, I'll put my hat in terms of thinking about Pillar. For me, the biggest benefits, or at least the value we've seen is ability to develop and test very reliably, very low latency platform. The other thing is the re resiliency of pillar to be able to do at this scale. It's unparalleled in my mind and these things, putting a CTO hat, are the ones I'm looking for on all my platforms. Anywhere where it makes sense for us in terms of applying this technology, we are leveraging it to the health.

Josh King:

Alexei, why is now the right time to hand off what you've done at the NYSE? How will it feel to let go of the super fast machine that you've helped to build?

Alexei Lebedev:

One way I would answer is by trying to maybe lessen the drama a little bit, because I'm not disappearing. I know the team that I've been working with very well, and I'm going to continue to interact with them. In a way, I'm not going to another exchange. I'm just going back to the simpler things that I have been doing and that I enjoy doing a lot, building the tools that then allow you to create an application. The reason it's the right time is, look I've been working on this for almost 10 years. A lot has happened in these 10 years. There are new things that you can build. We have four markets on Pillar and I just had to make a call at some point. I didn't want this to be three years from now. I didn't want to just continue going in the same exact direction for so long.

Josh King:

Pillar's in good hands though you think, as you get ready to move on to your next chapter?

Alexei Lebedev:

I think it's in very good hands. I could go on and on about all the software quality methods that we've innovated in this team and the great people that are on it. I don't doubt that it will perform solidly 10 years from now.

Josh King:

Mayur, as a whole, we know that ICE employs about 5,000 people around the world. How many engineers, doing what kinds of jobs keep this engine that we're talking about, building the exchange of the 21st century?

Mayur Kapani:

Let me just talk about it in general, from our technology perspective. We have about 1700 engineers overall. If you take our operations staff that monitors and manages the technology, we have about another 1500 to 2000 folks. We are a large technology organization. Our product is kind of manifests itself in technology in terms of all our products are listed on computers and all our products are access through computers. Our products are access through networks all over the world. Our whole product structure is completely realized in technology. Technology plays a major part in our product development, in our way, we serve our products to customers.

Josh King:

Where do you find the best people to join these ranks? What does it take to be able to be one of these technologists that can operate at the speed and resiliency and level of paranoia that you started talk about in the first place? Where do you find them?

Mayur Kapani:

It's a very interesting question. For me, the most people I find... I find people in many different areas and I want... There are people who are passionate about a particular technology and want to apply to this scale. Number one, we find those kinds of folks. The other folks are people who have a strong interest in the financial domain, want to understand how the markets work, want to understand how to make them work and want to put in that effort to learn about the technology to make it work. We find people who are deeply technical or deeply passionate about the markets, the ones who are most successful here. Our average tenure, if you look at across all our employees is about seven years. Most people who join us like to stay even from the acquisitions we've we have done in the past. It starts all the way from people with bachelors and master's degrees wanting to get into the financial field, to people with experience in other markets, especially in technology, and want to try their hand in up applying that technical knowledge to a problem of our scale.

Josh King:

In many industries, Alexei, it's survival of the fittest. What qualities would you look for in someone looking to carry on the work that you've done here at the NYSE?

Alexei Lebedev:

I think mainly it's just liking software. Software is complicated, and it requires a certain skillset that's independent of the industry that you're in. Because software is nothing more than some logical construct. As we all know, the more you add things, the more there are chances for things to break. What makes successful software is the same set of skills, regardless of where you are. Aerospace in exchange technology, or in web applications. It's some logical rigor and adherence to good principles and rules. I would say liking software, liking to build software is that criterion.

Josh King:

As you reflect on everything you've done here at the NYSE, looking back, what surprised you the most during various parts of your tenure?

Alexei Lebedev:

I'll give you an answer that is slightly off topic. When you enter the NYSE lobby, there is a little museum of sorts, which has machines like the original ticker plant, which is essentially a mechanical device, sort of a telegraph with some motor attached to it for printing quotes to a tape. There's also a Palm device, which was used on the floor to enter quotes and trades. What surprised me is that there is no physical manifestation of a modern piece of software. When someone enters that room, they can certainly see in some proximal sense, what were people thinking in the 19th century, but they can't see what they were thinking in the 21st century.

Alexei Lebedev:

One of the ideas that I've had, and maybe some of the listeners who have more time will take this idea up is write a program that reads the control flow graph a piece of software and generates a 3D model that corresponds to it, and then print this 3D model using one of the industrial 3D printing methods so that you can get a visual sense for a simple program like hello world, you would just get a simple printout and for more complicated model like let's say the New York Stock Exchange matcher, you will get something much bigger and more intricate. It will be very interesting just to see what it looks like.

Josh King:

Mayur, I assume you're hiring now. What kind of people are you looking for?

Mayur Kapani:

I think for us, there is a technology side of it in terms of there is some core skill sets we need for these platforms, which we need for low latency, like C, C++, or working on large data sets and working on platforms like Hadoop. From the technology standpoint, we are looking for those kinds of people. Then we are also have whole set of applications that are serving customers through the net and accessible through the net. Those again are Java technologies and technologies with HTML5. All these standard set of technologies where anybody who's interested, obviously we have a lot more job openings in that area. But more than the technology, we also would like people who really are hungry to learn about the markets and would like to solve problems to this scale. Clearly all these technologies are not unique in the sense they are used in a lot of industries, but the way they are applied in our kind of an environment, somebody needs to have that passion and wanting to do it to the scale and be part of this journey.

Josh King:

Sridhar, where do you get your best people from?

Sridhar Masam:

Again, to Mayur's point, I think, again, bachelors or masters in computer science or electronics is where we have success from. We have people from telecommunications background who have been pretty successful in our areas as well, and with financial background as well.

Josh King:

In any function, the top practitioners debate the key issues of the day. What are the hot button topics in technology right now in the circles that you guys travel in? What are people talking about?

Mayur Kapani:

At least in the world, I deal with people talk about how to deal with large amount of data. Specifically, not just taking the data and storing it somewhere, but how to actually use it in a meaningful way. Whether you call it AI machine learning, whether you call it big data tools to do ad hoc queries, that's one big area here in terms of what people are trying to do. The other area where people are trying to do in our circles is we've seen the Intel's or general decrease in CPUs clock cycles where every 18 months we used to see a CPU power double. That has slowed down considerably.

Mayur Kapani:

Now people have come up with much more creative ideas in building special chips that implement certain algorithms and how to paralyze those algorithms across large number core that are limited by the amount of cycles you can put on a core. That's one more area. One other area where we see a lot of innovation is bandwidth in terms of figuring out ways to push more and more through our pipes and creating bigger pipes. All these areas clearly are near and dear to our heart in terms of all the problems we need to solve. Those are the areas where at least in my world, the things I worry about and try to look for the next disruption.

Josh King:

Walking onto the trading floor, Sridhar, it's impossible to ignore the sound of electronic buzzers and beepers, even the constant hum of technology powering the machinery that runs our equities market. On average, NYSE group trades more than 1.7 billion shares per day, but make the case for our listeners as to why the NYSE is a tech company and not simply a financial organization?

Sridhar Masam:

At its core, NYSE is a technology organization, by the way, of how cost customers interact and trade with us. Every aspect of our trading systems is built on technology, be it auditor entry gateway, be it market data systems, be it web portals for customer to access, even the technology on the trading floor for the DMM, as well as floor brokers. It starts from a listed company from an IPO all the way the technology is for them to actually, do all the regulatory obligations as a listed company. Everything is provided through technology. There's a web portal called listings manager for listed companies to kind of fulfill their obligation. Every aspect of interaction with NYSE is through technology.

Josh King:

Mayur, one of your many duties is to lead technology diligence on the company's acquisitions. Last year, we completed our acquisition of fixed income marketplace, TMC Bonds, and trading platform BondPoint, which positions ICE as the leader in electronic trading in the fixed income market now called ICE Bonds. Can you take us through your part of the due diligence process on evaluating these potential acquisitions?

Mayur Kapani:

Sure. We've done a lot of acquisitions and ICE general business strategy hinges on finding companies that are a good fit, are complementing us in terms of a specific product or technology, and can be distributed to our network of customers. As you look at the two bond platforms we acquired last year, like in all other cases of acquisitions, the business side kind of convinces themselves that the product set is a good fit business strategy, and it fits it well into our business strategy. Once that due diligence is done, it comes to the technology side. On the technology side, we start looking at, okay, what are the key technologies used in the platform? What is the security posture? What is the skillsets of the teams that are working on those platforms? Then we form a view based on the platforms and skill sets and how we want the integrated product to look like from the customer standpoint, how we are going to evolve the platforms. That becomes the focus of the due diligence, figuring out all these elements so that they become part of the ICE technology family.

Josh King:

Generally speaking, that muscle for integrating these technologies and doing these acquisitions is far more evolved for you in 2018 than it was 12 years ago, 2006, when you first joined?

Mayur Kapani:

I think what really built our muscle was 2013 acquisition of New York Stock Exchange with the multitude of platforms. Obviously we tested it again in 2015, when we bought the IDC data platforms, which had a multitude of data platforms and technologies, which we needed to integrate and harmonize.

Josh King:

Just taking a quick trip back to 2012, 2013, you get on a plane from Atlanta, show up at this place at 11 Wall Street. What strikes you immediately about the challenge at hand?

Mayur Kapani:

You go into a place with 200 years of history, I started appreciating a lot more, what important role ICE was playing and to take care and be the custodian of such an institution. You had to be a very thoughtful and figure out not only in terms of the business side, how are we going to run the business, but also from the technology side, how are we going to integrate the technologies and the people such that we have no disruptions and we don't impact the markets in any significant ways.

Josh King:

And it still is job one. After the break, Mayur, Sridhar and Alexei and I talk about the latest technology inside 11 Wall Street and the tech advancements about to come down the pipeline as we build the exchange of the 21st century. That's right after this.

Speaker 7:

Spirit Airlines is a Ultra Low Cost Carrier in the United States that is intended to democratize travel, provide low fares and high quality service for the people in the continental 48 and Latin America. We offer a very low fare and in exchange for that, you get an on time performance that can't be beat. Our growth has really catapulted us to this point. We're extremely proud to be a partner with NYSE. Spirit Airlines, listed on the New York Stock Exchange.

Josh King:

Back now with Mayur Kapani, Sridhar Masam, and Alexei Lebedev, part of the team of masterminds behind ICE's technology infrastructure. We've been discussing building the exchange of the 21st century and the paths to ICE that Mayur, Sridhar and Alexei, each took to find themselves in a position to be doing just that. 2015, we mentioned before the break, ICE acquired financial market data and analytics provider, Interactive Data Corporation, IDC for $5.2 billion. You said that this marked a new chapter for technology and development at integration for ICE, how?

Mayur Kapani:

First is, traditionally, we were running exchanges and clearing houses. A lot of the data we generated was our own proprietary data, which we created channels for distribution. With IDC, we got into and before that, even that super derivatives in 2014, that was our first foreign into getting into a market data platform. We started becoming the source for lot of evaluations with IDC. These were fixed income evaluations with super derivatives, which we got in there were OTC derivatives pricing and evaluations. That was a new business we got into. The next big thing we did is we started consuming large amount of data from exchanges all over the world, aggregating it and creating all kinds of derived products and valuations and distributing it over many forms like APIs, files, through a messaging medium, where we would distribute these messages on the wires and customers would consume it. We got completely new business with much bigger footprint in terms of the sources and the delivery channels through which we distributed this data.

Josh King:

It's the availability of data that really is what all customers want. Ben Jackson, the President of Intercontinental Exchange and Lynn Martin, the President of ICE Data Services have said about the importance of meeting and exceeding customer demand on this podcast in earlier episodes. You've also stressed the importance of understanding and focusing on the needs of ICE's customers stating customers want and need data anytime, anywhere. It's important to be able to access data across multiple device. What's the process to recognize those needs and then integrate it into the technology that ICE is building out?

Mayur Kapani:

I think instead of saying devices, I would say multiple channels. Depending on who you are, if you are a direct customer that is trading a given product, you are a listed company who is looking for information on that company and everything that is happening in terms of news and transcripts and fundamentals and all that analysis. Customers have different needs. We work really hard to provide channels that is best suited to their needs, be it in the form of an API, be it in the form of a front end or a Gooey where the front end itself is delivered on their desktop or front end is delivered. Right Amount of information is delivered on their mobile.

Mayur Kapani:

There are cases where customers are looking only for a specific set of analytics. They're not looking for raw data. We try to do all these things where we slice and dice the data, identify the right delivery channels and make sure we can serve the customers with what they need in the most efficient way. We don't want to give them data more than what they need and make it expensive for them in terms of their bandwidth storage or any other means by which they consume the data. We go through this process of identifying the customer segments and figuring out what segment is looking for what data and how we can best serve that customer's segment.

Josh King:

The company acquired instant messaging platform, Yellow Jacket now operating as ICE Chat in 2008. Instrumental in its recent development is the use of artificial intelligence. How does an IM platform for traders incorporate AI into its platform and where else can AI be found in ICE or NYSE technology?

Mayur Kapani:

Yeah, so it's very interesting. When we acquired Yellow Jacket, one of the things that was appealing for us, it was actually started by two folks from MIT. It was a very good idea where they were taking in a lot of texts, which is basically they understood the lingo of the trade traders. Even in a given market segment and using that lingo, they started creating grids and also valuation models for options, where they would read and understand and learn all these texts that were being exchanged between the different traders and try to create some meaning out of it. For that, the technology behind the AI technology in terms of passing text is fairly well developed and was fairly well developed. But what they did unique was the fact that they converted that lingo into a structured set of instruments so that they could provide the traders with much more context. They could provide the traders where they could hook in those texts coming in to the valuation models that would give them a price.

Mayur Kapani:

It helped them really scale a given trader really scale in terms of number of courts and number of prices they could give to anybody who was looking to trade with them. There is a good example where we took that technology, used their algorithms or AI, enhanced it where we expanded the number of instruments significantly, where we took the outputs of some of those parsing recognition, we call it internally techs, and hooked up all our data sources so that we could give the traders much more contextual information of what was happening in that instrument. We also integrated this into a lot of our front-end tools like Verb ICE and ICE Block, where the listed markets and the chat markets are all integrated on one screen and helped traders and companies make much more effective decisions on how to price all these derivative instruments.

Mayur Kapani:

This is one good example of where we have leveraged AI to the hilt, another area where we are exploring even though we haven't had as much success and we continue to look at this is automatically parsing the bond prospectus and trying to find structured data in those bond prospectors, such that they can automatically create reference data for a new bond or a new issue, and use that as part of our reference data offering.

Josh King:

Sridhar, as the head of NYSE technology, you oversee the operations behind our exchanges and ensure that we're constantly updating and growing the technology that we provide to our brokers and traders. Now, I've heard some people describe the ambitious pillar project is like trying to change the tires of a bus that is careening at full speed down the auto bond without letting it slow down to a stop. Can you and Alexei, tell me about some of the challenges that the NYSE has faced in updating and introducing this new state-of-the-art technology platform?

Sridhar Masam:

When you say NYSE technology or trading systems, we have multiple applications to start with. We have audit entry or customer facing gateways. We have ticker plans, which is reading and market data from our markets. We have proprietary market data systems. We publish marketed data. We have reference data systems. We have command and control tools. We have web portals, they're all distinct technologies working together making a trading platform. In the early phases of Pillar upgrade was to replace the matching engines part. At high level it takes buy orders and sell orders and matches them at a given price and in disseminates the trades to the respective parties customer. Our core mission with Pillar was to actually start, how do we replace the core matching engine is still keeping all the other peripheral systems to minimize the impact on customers at the same time maximizing the efficiency of our new platform.

Sridhar Masam:

Some of the challenges which one is, keeping just everything else the same, and just the matching engine new. There's a compatibility issues or things between address. How are they compatible, how the new platform is compatible with existing peripheral systems? That's one key aspect. The performance of the new platform is much better that impacts the performance of the... it exposes the limitations of the peripheral systems. It's a combination of how you maintain that balance in rolling out a new system while integrating with an existing system. That's part of the challenges we had faced.

Josh King:

Alexei, I think it was your metaphor about changing the wheels of the bus moving full speed.

Alexei Lebedev:

I think the biggest challenge in doing something like a project of this size is simply things that have been running for a while, tend to have a lot of parts that are not necessarily documented. They're simply known to have worked yesterday. If you don't touch it, they're going to work today. But if you touch it, it might break. Nobody really knows where that sensitive part is that you're not supposed to touch. As you can imagine, with 1,000 such parts, it creates a real hurdle to replacing a legacy system. As far as I know, a project of this size has not been done ever in a financial industry.

Alexei Lebedev:

You usually have one company acquiring another and the other company has maybe better technology and then they just decided to replace A with B. But the piece-wise replacement of a very large established system with something new, to me, that was a surprise when I realized the scope of that. Eventually, I admitted that it's necessary to do it in this very slow piecewise way. But once it is in, it is so well established and there are so many people connected to it. There are so many things that the system does for various participants in the sense that they rely on it. That it's actually very satisfying to see these things being installed in and being operated.

Mayur Kapani:

One perspective I can add, we talked about pillar, but we also, if you look at in general how we upgrade our technologies, because our customers... We have a large network of customers that connect to all our platforms. These customers, we don't want to do a big bang and impact them in a negative way. They are relying on us to do their day-to-day business, be it the hedgers who are trying to use our derivatives markets or be the brokers and others who connect to us on the NYSE trading firms, who connect to us on the NYSE. They rely on us to run their business. We have to be very thoughtful in terms of what we change for them on any given time. We have to be also very thoughtful on making sure that we have done adequate testing before it goes live so that it doesn't materially impact them and also exercise a roll rollback or practice roll back in case something does go wrong.

Mayur Kapani:

All of these means that it puts a structural limitation on number of components we can change at any given time and also puts a limitation on how long it will take us to do some testing or a change before it goes live in production. To Sridhar's points and Alexei's points, this was the most challenging thing we did along with some of the other migrations like we did. We converted from a life platform to an ICE platform on the derivative side. In 2014, we did the launch of the NYSE cover. We went from an old platform to a new platform, few symbols at a time, communicating with customers to make sure that everybody was in sync with us and minimizing any kind of impact it had on the exchange or their business.

Josh King:

Sometimes real-world events from outside the building also intervene. I want to go through a day where like an unexpected news cycle can send the market's soaring or causing it to drop dramatically. I'm thinking of a day like November 9th, 2016. That's the day after president Trump was elected. Let's take a listen to how the trading floor was reacting that day.

Trish:

And count down to the closing bell, anchor, Liz Layman, who they're on the floor of the stock exchange and Liz, what are those guys telling you right now?

Liz:

This is really interesting. Isn't it, Trish? Because overnight, I'm sure you got the same calls. What's going to happen to my 401k. The futures are down 800 points. Well, it sure is interesting. Given a few hours to absorb all of the news, the markets are fine right now. We are looking at a gain of 229 points for the Dow Jones Industrials. More importantly, we are now less than 100 points away from an all time point close here at 18,562. All time high, 18,636. We're seeing real gains here in most sectors, healthcare, financials. Financials are hitting nine month highs right now, Trish. Air is coming out of gold. Of course, over night, gold spiked about 50 bucks. All the gold bags got excited, but right now things seem to be very positive, but I've been talking to traders here. They're saying it just looks a little too cute and buttoned up. You never know.[crosstalk 00:52:50]

Josh King:

18,000 seems like a lifetime ago Sridhar, now we are back into a period of volatility. Yet we're between 25,000 and 26,000. How do you ensure that markets can operate optimally every day, but even more importantly on a day like that, or the day after Brexit, when you know the messages are going to spike?

Sridhar Masam:

Our commitment to quality starts started early on in the development process where we emphasize on code quality, writing tests, holding new lines of code, code reviews, our continued integration pipeline for every code change we test four hours of tests within 100,000 test cases, stability test, failure to and performance test as part of the capacity and performance management in team. We actually stress test all the systems. We know the limits of all these systems, the performance degradation at a certain level, and how the systems react. We have two times, three times high watermark than peak hour observed. We configure, prepare our systems to handle three times, four times of peak observed and so that will be able to handle this wallet holidays without any hitch.

Josh King:

Alexei, you mentioned that in 2009 on a bet that you wrote the first proof of concept that would reliably transact messages in about 16 microseconds under load without losing messages, the rest they say was history.

Alexei Lebedev:

Yeah, that was an interesting time when I ran a small high-frequency trading firm. We traded in the U.S and on the Continental Europe and in London. The exchanges in Europe were, I would say in terms of their structure, they were ahead of the U.S because they had things like volatility auctions that we didn't get until LLD amendment. In terms of latency and velocity of trading, there were way behind. They were essentially database oriented systems where events could take 50 milliseconds to execute. They used to charge people to place an order like penny in penny out, but there were slowly realizing that something had to change that the U.S was really leading the revolution. They wanted to talk to high-frequency traders and find out what it is that they think. It's during one of these conversations that we had these sort of get togethers where they would ask like, "What would you guys like?"

Alexei Lebedev:

We would say, "Well, for starters, maybe don't charge us to place an order because maybe we're trying to make markets there. We want to place a lot of orders. If it costs us money, it's a non-starter." The other thing we talked about was latency. We said, why are you guys so slow? What is it that you're doing? Two messages come in, can't you figure out if they match or not? It was very clear that they just had no idea how to approach it. They had like a cluster of 200 windows machines running something that net or other their best idea for improvement was to upgrade to the next version of Microsoft Windows. I said, I guarantee you and I had no idea of whether that was true or not, because I had never written an exchange. As a high-frequency trader, basically, you're trying to hit and run.

Alexei Lebedev:

If you crash, it's okay. You recover, download the latest data from the exchange. If you're an exchange, very different story. You have to record everything that happened. You can have a situation where there is an outage, or let's say we unplug the computer at a strategic time. You can have a situation where you have reported that an event happened to one of the participants and then you don't know that it happened. As a trader, I've actually had situations like that with exchanges, which told me that not everything is perfect on the other side of the fence. This added to my desire to just fix that. I went back to the U.S after my trip. It was August and things were slow. I just spent two weeks writing this algorithm for transacting data.

Josh King:

Not only does the exchange need to operate perfectly, market participants also need to be able to plug in and interact with our system. To a non-techie person, this seems like a fraught proposition. How do you test the exchange systems and how do you ensure that there's minimal client downtime when you do it?

Alexei Lebedev:

Testing is actually a lot of fun. There is some perception in the software industry that real men write code, and then you have these testers. They didn't make it as programmers. They're going to test instead. They're going to just run the programs, but actually correctly testing your program is a very difficult challenge. The reason it's difficult is because to test the program, you want to prove that the program works. In order to prove that something works, you need to enumerate the entire state space of it, and then prove your assertion for every point in. It's impossible. The state space of an exchange is so large that there is no way you can enumerate it. It doesn't matter how much computing power you have. You're going to wait until the heat death of the university is still not going to be able to test it.

Alexei Lebedev:

It is an algorithmic challenge. Your resources are limited. You cannot really totally prove that it works, but you have to get some solid assurance. We do things like we put 500,000 orders on one side of the book and then send one 500,000 share order on the other side, that's going to match them all. That's like a mega fill. Or we try to just stop the book with millions of orders to see if any component and the test is going to crash under load, or we connect every single user in our test environment. Then every single user sends an order at the same time and we see what the latency spikes are. Those are the kinds of tests that we write. It's a lot of fun to try to bring down to your system.

Josh King:

Mayur, how do our operations folks feel when we pressure test and red team against our own system?

Mayur Kapani:

Oh, they love it because they don't have to wake up and there is a problem. To Alexei's point, one of the things we need to worry about is what are the extreme conditions where sometimes not intentional, something malfunctions on the customer systems or something malfunctions on our end that can bring down the whole system. For us thinking of those scenarios and doing this as part of the creative mental exercise to keep the system running. The other thing which Alexei didn't mention, which is a big part of our test processes, doing replays, where we capture all the data that is ever gone through our system across months and play a plate back at two or three times the rate at which it came in to exercise our platform and be reasonably confident that any change we have done, or anything we have done to the platform, adding a new feature or any of those things have not broken or anything that was already working.

Mayur Kapani:

Those are the kinds of things that add to our assurance cycle in terms of building new software. One thing to keep in mind, we have to be ultra conservative, but we can be ultra conservative and still make progress by having these processes and tools to exercise and test.

Sridhar Masam:

Just to adverse your minimal downtime, our software when we deploy our architecture, we have hardware redundancy as well as software redundancy, where there's a piece of hardware fails. You have a backup, you have a process, A, fails, you have its own backup readily available. We had our share of issues and we have learned from that and improve our testing tools. That playback is what I mentioned is one of the tools which helped us immensely. One of the things we always try to improve on is recovery time. How quickly or automatically we can recover from a failure so that we can minimize the impact. Over the last couple of years, we have had a pretty good high percentage of availability even with the increased volatility in the marketplace.

Josh King:

What's the biggest tech advancement and accomplishment during all of your times here that you've seen?

Mayur Kapani:

For me, the biggest advancement, first is, culturally having more people who understand mission critical software, building that muscle or skillset in terms of developing production grade mission critical software. That's the big cultural change. We've been able to, whatever we have learned in one part of the business, kind of globally inculcate that same culture across our acquisitions and any companies we have all over the world. That's one bit and the technological advancements evolution, thanks to Intel and thanks to the networking vendors, the Junipers, [inaudible 01:01:38] and others, Mellanox, all these vendors. We have taken all those advancements and applied them to our problems. As those technologies have evolved and brought in the price performance and other benefits, we've been right behind them in terms of adopting those technologies, because we have such high leverage when we can do something much more efficiently.

Alexei Lebedev:

For me, I would say it is seeing the scale-out approach work in the real world. Typically, or historically an exchange is some sort of loose collection of servers sitting in a data center, talking over a network. At NYSE, the exchange is really a tightly connected group of computers that form a mini supercomputer. That's really the best way of viewing it. We don't even view the connection between them as a network. We view it as an interconnect, which is there's some technical difference in terms of the amount of routing or the amount of logic or hot plug ability that you may have there. But basically to get 1500 CPU's working as a single unit is the most interesting problem. It's pretty impressive.

Sridhar Masam:

To add to that, I think in terms of software development methodologies with agile and iterative approach, automation and tools, continuous integration and continuous delivery, the speed of delivery, time to market, the methodology which help us, there's a big change how things were and how we are now, in addition to the technological changes, which Alexei and Mayur mentioned.

Josh King:

The world is increasingly going open source. I know that given the intense security and scale of Intercontinental Exchange in the exchange business generally, that's a long road to travel from here to there. But what elements of the open source community make this a place where some of the most creative engineers can operate?

Mayur Kapani:

Like everybody else, we are big users of open-source, be it the Linux operating system that runs on commodity hardware to using all the standard web toolbox of HTML5 and other technologies, including Java for building out our platforms. We also on occasion contributed back to open-source based on some of the tools we have used, but being highly regulated and running in a mission critical, secure environment, we have to be very thoughtful and careful in terms of the tools we use, how we evaluate them and how we manage them. We embrace open-source, but are much more thoughtful and cautious about it.

Josh King:

As we wrap up gentlemen, running the well-oiled machine that is ICE Technology requires an expert team of engineers, computer scientists, data analysts, and more. Going back to our discussion on what makes the NYSE a tech company, What's your pitch to listeners who might be interested in joining the team?

Mayur Kapani:

My pitch is we run 12 exchanges, six clearing houses around the world's premier data platform. All of our products manifest itself in technology. If you want to build a scale-out mission, critical piece of software, definitely think of us. We are one of the best in the world.

Sridhar Masam:

Yeah. Just to add it, for building the next generation trading platform, which is highly scalable, high capacity, low latency, highly deterministic, the magnitude and scale of which is unparalleled with the cutting edge technologies and best engineering practices focusing on automation.

Alexei Lebedev:

I would simply say that it's the highest tech project that you could be working on if you went out to look for one, because the combination of real-time requirements uptime and the sheer low latency and scale of it is not to be found anywhere else. For instance, web applications typically operate 10,000 times slower than we do on the transaction size. That's my pitch. If you want high tech, this is what you want.

Josh King:

Even though the process may operate a lot slower than the highest high-frequency trading at the NYSE, if you're interested, go to standard old internet IntercontinentalExchange.com/careers and find out all of the opportunities that are available at ICE. The NYSE are 12 exchanges in six clearing houses around the world. Thank you so much. Mayur, Sridhar, Alexei for joining us inside the ICE House.

Mayur Kapani:

Thank you Josh.

Alexei Lebedev:

Thanks.

Sridhar Masam:

Thank you for having us.

Josh King:

That's our conversation for this week. Our guests were Intercontinental Exchanges, Chief Technology Officer Mayur Kapani, the New York Stock Exchange's Head of Technology Sridhar Masam, and the New York Stock Exchange's Chief Software Architect Alexei Lebedev. If you liked what you heard, please rate us on iTunes so other folks know where to find us. If you've got a comment or a question, you'd like one of our experts to tackle on a future show, email us @icehouseattheice.com or tweet at us at NYSE. Our show's produced by Theresa DeLuca, Kristen Coz and Pete Asch with production assistance from Ian Wolf. I'm Josh King, your host signing off from the library of the New York Stock Exchange. Thanks for listening. Talk to you next week.

Speaker 1:

The information contained in this podcast was obtained in part from publicly available sources and not independently verified, neither ICE, nor his affiliates make any representations or warranties, express or implied as to the accuracy or completeness of the information and do not sponsor, approve, or endorse any of the content here in. All of which is presented solely for informational and educational purposes. Nothing here in constitutes an offer to sell a solicitation of an offer to buy any security or a recommendation of any security or trading practice. Some portions of the proceeding conversation may have been edited for the purpose of LinkedIn clarity.

Listen to more episodes

Subscribe to Inside the ICE House

  • Listen on Apple Podcasts
  • Spotify
  • Listen on Stitcher
  • Listen on Google Podcasts
  • Listen on Himalaya
  • RSS

Information contained in this podcast was obtained in part from publicly available sources, and not independently verified. Neither ICE nor its affiliates make any representations or warranties, express or implied, as to the accuracy or completeness of the information and do not sponsor, approve, or endorse any of the content herein, all of which is presented solely for informational and educational purposes. Nothing herein constitutes an offer to sell, a solicitation of an offer to buy any security, or a recommendation of any security or trading practice.