Yeah, so Nicola, welcome to the show.
Uh, this is the fourth episode of Pale
Blue Nexus and um,
uh, happy to to have you on the show. We
met uh, prior to your big announcement
last week. Um, and uh, it was quite a
big announcement uh, let me say. It uh,
uh, something I knew something was
coming when I saw some of your your pull
requests on the open claw and I was like
I was really excited about that. So so
tell me about uh, your journey so far
and um,
about yourself and about the big race.
>> Sure. Thanks thanks for having me. It's
it's pleasure to be here. So
um,
I
I I started Deep Mind for about four
years ago with with two close friends of
mine. We we both met
while working at a previous uh, company
for a long time together and so
uh,
we really saw this need uh, as this AI
models were coming out.
The the interesting part about them is
they're just a computer application but
they're
they're like a computer program but
they're quite different. They require
this large amounts of
compute operations to produce even a
single token.
So the big models basically need
hundreds of billions
of multiplications to generate a single
token or a single word word out of them.
And so
we we saw these models kind of
just showing up slightly on the horizon.
It was 2022. It was before chat GPT but
you know, you could see for example
stable diffusion was this super popular
open source models that can draw
pictures. They had a lot of
artifacts as you remember like people
had a lot of fingers.
>> Yeah.
>> Uh
but it was kind of a demo of an open
source model that's quite powerful and
and showing something that wasn't done
before.
And at the time we didn't have chat GPT
so you couldn't chat with the model but
you could
essentially prompt the model to
to continue a sentence. For example,
instead of asking like what is the
capital of Paris, you had to say the
capital of Paris is
and let the model just continue the
sentence.
And even in that mode
you could still see kind of
signs of intelligence in the model.
Um
so we wanted to use our experience from
our previous
um I guess
company where we worked.
Uh we we built the infrastructure and
the back end for a messaging app with
over 200 million monthly active users.
And so we had a lot of experience of
how to procure and set up physical
server infrastructure
in data centers
and how to build like very
efficient kind of back end APIs on top
of it and make sure we use like the
infrastructure really fully. Like part
of our secret sauce for the messenger
was that we managed to
get a very high utilization on all the
hardware we bought
uh by mixing different types of
applications, you know, compute hungry
applications, memory hungry applications
onto the same machines
and balancing the load until we get
really high utilization of the
infrastructure. And so, I think
the idea behind deep infra is somewhat
similar. Like, we
building a cloud hosting platform for
these AI models.
Um
Again, these are kind of like new
applications. So, they need somewhere to
run.
And running them is what's called
inference.
Yeah. Um I can go more into our journey
like from very early
>> It sounds like
the challenge is that you solved at
IMO.im
are really sort of
an input into the same problem that
you're having with inference.
>> I I think so. Like, I think
you know, life is kind of a journey. You
see certain things that help you
kind of navigate the next stage in a
different way. And And it I'm all
the one of the main takeaways was like
how expensive AWS and the other public
clouds internet connectivity is.
And how
essentially more efficient you could be
if you do it yourself. But, you know,
it's not easy to do it yourself. That's
why people, you know, opt up for the
clouds because it's you get something
done and ready.
So, you need to invest in it. You need
the expertise to do it.
But, you can get to a lower like cost.
>> And talk about like the difference in
cost between somebody choosing like a
hyperscaler
uh solution like Amazon or uh
um Azure
against your solution.
>> Yeah.
Um
I believe like we we host a lot of
different open source models and each of
them has its own price. So it's really
not that easy to compare one to one, but
if you look at the GPU pricing,
you know, we are offering B200s at 275
and I believe like the public clouds
pricing are
in the 6 to 10 plus range.
And you typically to get that numbers,
you have to do like longer term
commitments as well and
and and commit to
to that cloud.
Um
And then on the tokens, there's another
layer of efficiency, right? Like we
believe we have really
efficient systems that can generate more
tokens an hour per card. And so that's
why we can offer
better pricing per token and also cash
token pricing that's really really
attractive. That's sometimes
10 times cheaper than than other options
or closed source models.
In summary, I think the big clouds have
really focused more on like hosting
Anthropic and Open AI models or Gemini.
So the real comparison here is like how
much does
I guess uh
you know, Anthropic's model cost
AWS versus how much does Deep Seek cost
with us. It's not the same model. Deep
Seek is still
slightly worse, like 3 to 5%
in in some benchmarks,
but it could be like 10 times cheaper.
>> Okay.
Um so cheaper is one thing. The one
thing I did notice though, I I I sort of
use your back end because I use Ollama
and you guys help them
uh provide their services.
Um I noticed over the weekend it got a
lot faster. Like I would type something
in and my my bot is responding like
right away versus there being some maybe
two three seconds in between when it
when it I don't know what happened
there, but uh keep up the good work.
>> I I I can share something. We
basically over the weekend like a big
batch of our infrastructure, new B300
GPUs came online. And so
there was a lot more breathing room for
all the models to scale up and and go
faster. I hope it's that. Uh
but
>> Whatever you did work, it's it's
uh it's responding quite faster for me.
So, um I can't wait to get out there and
use it. I feel like um that was a
meaningful upgrade uh from a user
experience perspective, so.
>> Thank you.
>> Um so, let's talk about scaling to
I guess your your app scale to 200
million monthly active users. And
this sort of inference level, how much
customers are you serving right now at
Deep Infra? And how much tokens are you
delivering?
>> So, at Deep Infra we're doing over 5
trillion tokens a week across like all
the models that we serve.
Um
we also do some
you know, cluster rentals to new labs
and and other customers. That's kind of
in addition to the inference business.
Um
it's been an interesting journey like
over the last uh basically four years.
It really started the growth really
started when Llama 2 came out.
And that was like the first commercially
allowed open source model that was
decent right?
For its time it was like the best open
source model. It was quite large. 70
billion parameters is
is
a was a big number at the time.
And so from that moment on, we just
started seeing growth in tokens. And it
was always
growing exponentially, but we started
from a very small kind of base.
Um
And so, then there was kind of like a
wave of new models coming out and us
raising more capital and investing in
more infrastructure. So,
for the Llama 2, we were only we only
had, I think,
couple H100 GPUs. Mostly it was A100.
And And then we got our seed round, we
got a bunch of H100s with that.
Then we could serve the Llama well. And
then
Me- Mistral came. It was like a small 7B
model. Then
And actually, one of the biggest changes
for us was when
um Mistral came. This was like the first
mixture of expert open-source model.
So, it's kind of like a new generation,
uh
think.
And
the story there was we were
I I I'll go a little technical. So, this
model was basically eight 7B models
together.
The experts were 7 billion parameters.
And
during runtime, you would use two of the
experts. So, the model should be as
expensive as 14 billion parameter model
to run.
But as soon as that model shipped,
everybody in the market priced the model
as if it's like
50-60 billion parameter model, like
using the full number of weights.
And we were really aggressive. Like to
us, it was quite clear that this should
cost as much as like a 14 billion
parameter model, because that's what you
would use during
>> runtime.
>> runtime. And so, we just priced this
model really aggressively. I remember,
you know, we announced 27 cents per
million tokens.
And at the time, you know, other people
were at $60 60 cents. And
>> Wow.
>> So, that got a lot of attention on
Twitter and a lot of traffic for our
model.
Um
I think around that time people started
saying "Oh inference
this is race to the bottom.
Look at, you know, the prices of these
things going down."
The interesting thing is that
even though this price was really
aggressive for day one, because when the
model comes up day one, it's not really
efficient.
It really, you know, over the next month
or two, like the efficiency of how well
we run this model really improved
dramatically. And
you know,
this model ran for like at least
year or maybe 18 months before we kind
of deprecated it.
Um I think the price went even lower
like after that. Like we just had more
and more efficiency and we could serve
it for
considerably less. I think
we just the software got better
on our side and
and so we could offer lower prices to
our customers.
>> What's the main factor there? Is it the
quantization or is it the the inference
sort of routing and engine?
>> It's a good question. Actually, there's
like a bunch of things. So, you don't
have to
>> to say your secret sauce.
>> Well, I I
I'm going to like mention like few of
the main techniques that I think make
these models more efficient that
um
I I'm happy to share. So, so
quantization is obviously one. And
interestingly, like the newer
generations of Nvidia GPUs basically
support low and lower precision math.
Um
we believe that and Nvidia also strongly
believes this because they're dedicating
large portions of the chip to this like
low precision compute. They're mostly
investing in this
4-bit
uh types. They're FP4 and VFP4.
So, so one thing is quantization of the
model which
to some of your listeners just means
like
you're compressing the model down. Like
uh
instead of using
16 bits per parameter, you use like
eight or even four. And that really
helps like make the model smaller.
You can both do more math the smaller
the precision is, but you can also
fit more
fit the model into like fewer cards.
Or leave more space for KV caching. Now,
KV caching is the other really kind of
important optimization.
In these agentic workloads,
what the model really does is it
basically sends roughly the same request
many, many times with like a little bit
of additional information at the end.
So, if you can
save some computation from the last time
you did the inference,
then the next time you have to do
inference for the same kind of customer,
you can be really efficient. Like you
don't have to bother the GPUs with the
big portion of the request.
And that's kind of a a big efficiency
gain. So, we were one of the first
people to have cached token pricing on
open router.
>> Okay.
>> And offer it to like our customers
obviously as well.
Uh
And that's a big
part of it.
And there's a bunch of other techniques.
And it's kind of all layered together.
And also more efficient kernels, like
moving to more efficient GPUs.
Uh
>> This sounds more like fine-tuning a
Ferrari than it does uh racing to the
bottom uh like type of thing, right? But
uh
it's a bit of both.
>> Yeah. You know, my analogy is I think
it's a lot like Formula 1.
And I'm a big fan of Formula 1. The
you know, the teams there are always
obsessed with like reducing the weight
of the car and then squeezing some more
power from the engine and squeezing some
more aerodynamic efficiency.
So like
at the end of the day you have to pursue
like all avenues of making this more
efficient
to kind of
end up winning or or being the top
few people.
>> Yeah. Um
and you know, you guys were
took capital from Nvidia. Um tell me
about like what they're doing uh with
Nvidia Dynamo. Is that part of your
stock now or is it going to be? Uh
and how do
And we'll get into the open source
versus closed stuff stuff uh stuff
later, but uh
tell me about Nvidia Dynamo right now.
>> Nvidia Dynamo is like
um open source project started by Nvidia
that is um
really like I think the next layer level
of efficiency in terms of inference and
scale.
>> Okay.
>> So,
I think internally at Deep Infusion
we've implemented large portions of the
ideas behind this already and we're
working more and more closely with the
Nvidia team on
on on this project
as well.
Um
For some of your viewers, it's basically
something like
um
uh kind of like Kubernetes in terms of
it helps organize how the inference
happens because inferences
to get the ultimate efficiency, you you
need to run this kind of at a big scale
and and Dynamo kind of helps organize
the scale around us and disaggregates
like one portion of the work with
another portion of the work.
So,
I think it's an important project.
Uh
and I think Nvidia is developing it
really well.
We essentially supporting various
different inference engines underneath
like VLLM and SGLang in addition to the
homegrown TensorRT-LLM from Nvidia.
I think also the project supports also
non-Nvidia hardware, which is
I think the right approach to make this
the real fundamental system for for
inference. So,
we're generally excited to collaborate
with Nvidia on on this front. And you
know, the relationship with Nvidia
started more on the technical side. Like
we
see a lot of tokens going through Deep
Infra and we see a lot of tokens flowing
through like various generations of
Nvidia GPUs.
We were really early adopters of
Blackwell
>> All right.
>> type GPUs. And so,
whenever we see any
instabilities, any crashes, any
performance issues,
we would share this with the Nvidia
technical teams. And
uh on their side, the Nvidia technical
teams would help
uh kind of address and fix and improve
I guess the libraries
that we use for inference.
And so, it was a good loop. Like, you
know, we would
share like important information that
helps kind of make
decisions on their end of like what
models to support and
what models to make more efficient and
and what models to fix like stability
issues.
And because we have a lot of traffic, we
see all these random
I guess
um events that cause like weird crashes
that are really hard for
for someone who who doesn't have the
same amount of traffic going through
their system to to notice and and fix
and improve.
So, we're just pretty open and shared a
lot with the team and
I think
I'm going to speak in a little bit for
them, but like for Nvidia it's really
important to have really efficient
software for inference, right? Like at
the end of the day like the their
customer measure whether to buy their
hardware versus someone else's hardware
on like
how many tokens can this thing generate
for me in an hour?
And does it have like support for many
different models that I might want to
use?
And so
I think they're coming to this with the
right mind. They understand that they
can't just like
sell the GPUs and that's it. They need
to invest a lot of engineers to work on
the software layers of the stack to make
the GPUs really
useful to someone.
Not just us, like just any any any of
their customers, right?
So
I think they understand this well, so
they've invested a lot of effort into
the software side of their cards
to basically show like the highest
inference performance um
compared to
other hardware.
>> And I mean, this reminds me of a few
different parallels within the tech
world in the last 20 years. One of it
being like
the
um the latency aspect. And the other
part is the open source aspect, like Red
Hat. But going back to latency,
um
uh you remember how high-frequency
trading used to happen closer to New
York where the data centers were to to
to the exchanges. Um are you
experiencing similar
uh build-out challenges where people
want higher lower latency? So, they want
to be closer to their cloud Uh or or to
their infrastructure.
>> E E both kind of yes and no. I feel like
the answer for inference is more
complicated like um
Let's take a typical request, right?
Like imagine you asking ChatGPT to do
something and and passing like a big
document together like with your initial
prompt.
So
the model needs to do this initial step
of like
look for all the data you've given it
before it starts giving you the first
token. That's what's called the prefill
phase.
And so that might take like from
100 to 200 milliseconds all the way to
like seconds depending on how much data
you've given it.
And then the model starts giving you
tokens and
and then like it might take 5 6 seconds
for the whole thing to complete, right?
Like the model starts giving you a text.
And so
actually there's a lot of latency
building to the whole inference like
from end to end the requests take like
long time like on the order of 5 seconds
to 10 to 30.
And so we've noticed that
we don't believe like
trying to put the actual compute like
right next to the humans
is the most efficient way.
Like we
to get lower latency, I think we just
have to build like the right
orchestration of GPUs and hold the model
in a way where we could generate the
tokens well and do the caching well.
And the other thing is like
I think if you look at high frequency
trading, they need like the very latest
data from the exchange and they need to
run like something really quick and like
make back like
a request to buy sell.
And uh
all these things are super small and and
they don't need
to run a super complicated algorithm
to decide like which of these two ways
to do it. So, they don't need a lot of
power is what I'm trying to say, but
inference is really, really power hungry
and compute hungry. That is why like
you know, there's all these data center
projects in the US and and most of them
happen in like rural areas
not near the city. Just the price of
electricity and like the
economics of deploying the
the GPUs work best in like
more rural areas. So, what latency is a
big portion of it, I don't think the
solution is
mostly the solution is not let's move
the GPUs next to like the population
centers. There will be some sort of GPUs
nearby for like very low latency tasks
like voice
>> Mhm.
>> tasks
or maybe even robotics tasks. Like there
will be some things that would require
ultra low latency.
But I think the
you know
the AI engineer, software engineer, the
AI doctor, the AI
um teacher potentially
the the latency would be okay for these
things to be hosted from
just a AI factory in in in the middle of
the country.
Now
I think you had another question about
the open source models. I'm happy to to
dive into that. Like this is a big
theme.
>> Yeah, I mean, open source I am a big
Unix fan. I I started my career being a
Unix administrator.
Um
and I've just seen how it's been
proliferated throughout the years uh
with Red Hat going to enterprise
um and different flavors of Unix. Even
our my Mac is running on a flavor of
Unix right now. So, uh talk to me about
open source and and how that's a
strategic sort of like
bet for you guys.
>> Yeah, I want to give you like a little
history of myself, too. So, I'm also a
very big
user and proponent of this since I grew
up like one of my
teachers in
in high school that that taught me how
to to program, he was really big
proponent of of Linux and Unix. So, like
I've used Ubuntu since
the
basically 2004
and ever since I've always like had
Ubuntu like as my operating system at
home.
And even I want to tell you a story on
the phone side. Like whenever like
iPhone came out, right? Like Google
Android operating system was supposed to
be really open source and
open source. I'm still sticking with
Android since like
the beginning.
Although I have to say like it didn't
turn out as I hoped it would.
I feel like sometimes
I don't know. Android doesn't feel as
open source as
as it should.
But
on the topic of open source versus
closed source models, it's kind of a
really serious discussion that
I think
when we started the company Deep Infrah
in 2022, that was the biggest risk to
us. Like I knew that there'll be a lot
of demand for inference.
I just didn't know if they would be open
source models. And it's a tough job
because you have to invest like
considerable amount of computer
resources to build a model.
And then you make it open.
And so like not that many can do this.
Like it requires support from
big organizations. But any open source
thing has like
all this community innovation that
happens around it. And so
um
and it in the innovation feeds off of
each other. So like Facebook made Llama
or Meta made the Llama models.
And
then you know
Mistral did the first like mixture of
expert ones.
Then Deep Seek really took this to the
next level and did a lot of small
experts and added like some interesting
attention things.
Like the MLA.
And so like
then a bunch of other people basically
managed to train Deep Seek like models.
Like Chime and GLM and
And so like I think the
cool ideas get adopted.
And some more computer resources getting
invested into it. So
I feel like AI is really important. It's
good for us to have decent even if
they're not the top open source models.
It's still I think a good option for
enterprises and
just researchers and universities and
all of us as a community to have a good
layer of decent open source models we
could build on top of.
Um over the last 3 years as I've been
watching this develop
I believe like the o- the closed source
models have improved dramatically but
the open source models have really kind
of caught up to them. Like when when
Llama 2 was out and OpenAI had their I
think GPT 3.5 or 4 at the time
the gap was really big, like
you know, almost half the intelligence
or something like that based on like
benchmarks.
But now, if you look at how well they
stack, still the top models in terms of
intelligence are the the top three
closed-source lines of models, but
basically the next right after that are
open-source models. I wish
we get more US-based open-source models,
and I think that's coming
thanks to Indian in some sense for doing
NeMo Tron
>> Mhm.
>> line of open-source models, but also,
you know, we believe like
a lot of other labs in the US will also
release open-source versions
of models as well.
So,
I
Yeah.
>> My stack is like I were I was using
the frontier models, the US frontier
models, and then I just found one day
with Kimmy 2.6, it was like, I can't see
the difference anymore. It's It's almost
It's not 100%, but um it's caught up
quite a bit, and with Deep Seek 4
uh Pro and Flash, it uh
it's it's the edge is going pretty fast,
so
um I think like for me and maybe a lot
of other devs out there that are looking
at this stuff, it's it's coming closer
and closer. So, we can see you know,
what happens 6 months from now, right?
So,
uh there's that as well.
>> I agree. And I think another thing I
want to point out here is
even for a moment as a mental exercise,
imagine we freeze all the models right
now. No one makes any models any better.
The exciting part is like
we've gotten to a level that is really
useful.
Like even
expert software engineers, and I like to
like I'm one of them
are mostly not writing any code anymore
like they're using this agents, coding
agents and this
frontier style models both open and
closed to
produce
much more kind of software than they
could otherwise
uh by doing it by hand. And so it's kind
of like as if we've invented the
compilers.
And from now on we'll just write
high-level code and compile down to like
the machine instructions. I feel like
the AI models are kind of like a
compiler for us. Like we give high-level
instructions and the thing lowers it
down to
code that then gets lowered by the
compiler to the actual machine
instructions.
>> Yeah, you're right. Like this
spec-driven development type of
uh idea where you can just give it a
spec uh and ask maybe the frontier model
to come up with a spec. Then you can
take that spec and give it to a good
enough model and it would as long as it
has some guardrails it'll go and code it
for you. So.
>> Yeah.
Exciting times we live in.
>> Yeah. Yeah, you could really boot up a
lot these days. Talk to me about like um
the next level of security and
compliance as as these sort of solutions
get rolled out to to to bigger companies
and enterprises and bigger businesses.
Like right now you guys are SOC 2
compliant and ISO 27001.
Um
how are you seeing are you seeing any
pushback from bigger enterprises? Um are
you looking to go to the next level on
on that that that regulation front?
>> Um on the compliance side there's
we
First we always like our posture in
general in terms of us owning and
operating the actual underlying kind of
infrastructure and having like
almost no like sub processors underneath
us only like very rare cases. So, most
of the tokens that we generate happen on
our own
hardware in our kind of own spaces.
Um and I think that's kind of a
what enterprises should be looking for
um just to understand you need to
understand well the chain of who is
going to handle your request down to the
actual execution. You want to make sure
you're not like giving it to someone
that will give it to someone that will
give it to someone and maybe the final
inference happens in China.
Right? And so I think companies need to
be careful about like who does the
inference, where, and like whether they
trust this company.
So, as a startup it's a it's a journey,
right? Like we can't convince day one
like some super large enterprises to
to like trust us with their data, but
we're definitely
far along on this journey and we have
like large enterprise customers using
our infrastructure.
Now,
I actually see this as an important
thing. Like even though the the models
are
some of them are trained outside of the
US, we're kind of like the safe
environment where we run them on our
US-based
V100 Nvidia GPUs and
we have zero retention policy, so we
don't store any data.
Um
kind of like this zero retention thing
because
you know, you don't have to spend
anything on storage and your customers
feel safe and and you have this less
exposure, less risk of
you know, stuff leaking out out of your
infrastructure.
So,
I think
the security, by the way, of this models
is going to be like probably the biggest
thing we will
be thinking about in the future.
I think it's not as much about like it's
not in the traditional sense of like are
my, you know, data center secure? Is my
system secure? It's like a little bit
about the model itself.
And there's a lot more verification that
we need to do to make sure the model
wouldn't
make a mistake that might be costly
or cannot
you know, all these prompt injections
are basically us kind of
tricking the model into doing the thing
that it shouldn't be doing. So, there's
a lot
I think we're not really safe there on
many fronts still, even the frontier
models. Like I
um
I'll give you a small example. Like I
think we're still not safe in the way
where you can just say open claw, here
read all my email.
Cuz I feel like the right scammer would
basically send the right email
that will prompt inject
the model potentially to share
like information and take over. So, so
some of these things are
we need to be careful with, but on the
other hand it's kind of
you cannot be too slow in adopting this
technology because then your business
gets kind of left behind
>> Mhm.
>> or your competitors are taking the full
advantage of it. So, it's we need to
balance the adoption.
>> Mhm.
>> I think at the moment there's like
definitely risk, but
I think the rewards are still higher.
And so, you have to take some of the
risks. You just have to minimize the the
risk as much as you can.
>> Tell me about like Nemo claw. Like I'm
very interested in this topic. Um it
sounds to to like there's obviously like
a wrapper on top to help make open claw
more enterprise friendly.
So, I kind of see where that's going.
>> So, I want to explain to like the
viewers about
how I think these two things work out
like you can either have a very
There's two ways to build these things.
You can either start with a system
that's really open and is allowed to do
everything.
And then just see what are the bad
things it could do and try to like
prevent some of them.
And I think that's more of a open claw.
Like by default the thing is
really open. It can like
do many things.
While Nemo claw is Nvidia's kind of
attempted
starting the other way like start with
something really secure. Like by default
the model is not really allowed to like
access the internet or
you know, do many many things or execute
anything.
You have to explicitly allow what the
model can execute and access and
basically that way you can
start from this end. Like we start with
something really secure.
See what it actually needs. Only allow
the minimum set of things it needs to
really execute.
And that I think makes sense for
obviously enterprise adoption.
Um
So, I think there's active development
there.
Uh we
will see where this goes, but
we're looking you know, more and more in
kind of ways to encapsulate and secure
these agents. Give them enough freedom
so they can execute, but also make sure
that they don't have
um unrestricted access.
Uh
and
and then also I think more and more
there will be work that essentially
monitors the work of the agent
and looks for
essentially suspicious activity that
uh
might happen and and then you can secure
the agent in that way as well.
>> Okay, that's interesting. So so that's
that's the where NeMo Guard comes in.
It's like uh a
trusted sort of open guard in a way.
>> Yeah, it's basically open guard inside
like uh uh for lack of better words like
uh a jail or a shell.
And so
before it can do anything, it has to
ask.
>> Okay.
>> Well, well, you can you can see what
it's trying to do and then you can
decide to approve some of these actions
and and build this more contained
environment for it.
>> Okay. Um let's um talk about the the ICP
for
DeepMind infra. Like, are you guys
expanding into the
the more regulated space, the the uh
I guess the enterprise space? Are you
happy being in that uh small to medium
business, startups, tech startups area
where where your cost savings are much
needed?
>> The way I look at it is I feel
the demand for tokens is
really high and it's coming from
everywhere.
>> Mhm.
>> And
we
don't have
it it's kind of in stages.
We really want to help the people that
need
a lot of tokens at scale.
And at the moment, I feel it's not
really the large enterprises. They're
still figuring out how to use it and
where to use it.
It's the people who really need tokens
at scale are like
kind of AI first companies that are
building a product that it's like
AI powered from the bottom up. Not
Um you know, how do we insert some AI
features into our existing SaaS, which
still is quite powerful use case.
So
We we we try not to lock ourselves in
any segment. One thing I can say is
probably we're not a best fit for like
the super regulated people like banks
and
>> Mhm.
>> other folks are really interested in
doing this like in a extra secure way on
premise.
And that's
you know, our inference cloud is not the
best fit at the moment for this use
cases.
Uh
So yeah, we just try to scale our
platform, add more compute resources to
it, add more models, and then
Um we have a very light outbound motion
at the moment. Mostly all our clients
are kind of inbound.
And
so we we look to help the people that
need tokens the most.
>> Okay.
Tell me about like how you've been
handling these last few weeks personally
in your life. Uh it must have been quite
a whirlwind
uh over the last month or two. Um
how how are things going and how are you
balancing that? Are you going to the
beach or having some fun?
>> It's really tough, I think.
One thing I can prepare your viewers for
like if you want to be a founder
you know, you're you know, doing it
because you're hoping that the company
is successful, but when the company is
successful, then the amount of
work for you really grows a lot and
you kind of have to
your job is a little bit of more
building the company than building the
actual product. Like you have to go and
find the right people and hand them off
each like particular area,
and trust them, and
So, there's a
a crazy amount of work right now that
leaves no slot empty on the calendar and
no time to
cover everything. So, you
And you have to just try to think every
day about like what are the most
important things for us today and this
week,
and prioritize those versus like the
incoming like
wave of tasks and other things that are
coming to you.
Um and just
just move forward.
Um
>> What kind of culture are you going to
bring to to DP Infra internally?
>> So, I'm really happy that I got to
recruit a lot of my old team from from
my old messenger where I worked together
with some of these folks for many years.
Mhm. So, you already
you we we already had like a lot of work
together and and culture
together. And so,
you I don't like scaling the company too
quickly cuz I want to make sure like the
new people get to learn from the
existing people about like what we think
is
the right way to think about problems
and approach them.
And I don't
think some companies grow too quickly
and then there's not enough scope for
the individual people to do. And they're
working on things that are not
really high priority. I think it's
better to be a little bit understaffed
>> Mhm.
>> than overstaffed.
Um
I mean, culture is pretty important. We
We're very light. I don't think we have
any managers, really, at the moment. So,
that's expected from a small company.
>> Cool. And uh so, what are you looking
forward for for the rest of 2026? Like
how how do you What's exciting to you?
>> I think I'm excited about scaling the
compute clusters.
So, there's like a crazy build schedule
over the next few months, and
things going to get even crazier towards
the end of the year. Uh
I guess bringing these clusters up that
we've already kind of in the middle of
the production.
Just to give you an idea, by the way,
like setting some of these compute
clusters is like I think a major
construction project. Like
uh
you know, they cost like
tens of millions of dollars.
And there's a lot of physical setup for
them.
You know, electricity wires, like
it's kind of almost like building a
house. They need like containment walls
and
and then cabling.
So,
the That's exciting. I think I'm excited
to scale the company. Like we
are actively hiring for number of
different positions and and
interviewing. And so, getting
the right really motivated, experienced,
but also hungry people
to the right positions and giving them a
chance to just like contribute is really
important. I remember like in our
journey, like we
had
quite a few kind of key hires. Like we
couldn't have gotten here without
getting
this person to join us right at this
moment, and they really took a big
burden out of what I was doing before.
And they kind of well. So, like that's
uh scaling the
the team
is an important part.
I think we
Our business is really capital
intensive.
>> Mhm.
>> So,
for us to be successful, we need to be
good at raising funding, both equity and
debt.
>> Mhm.
>> Debt is an important vehicle to finance
these clusters and
this this infrastructure, and you have
to
I guess
do a decent amount of it, and you have
to do it really well
to get to like better cost per capital
cost per token, as well.
>> So,
exciting times.
Um
Just a random question. Are are the
the servers the
the hardware that you buy,
are they
do they have some level of depreciation
over time?
>> Yeah, you know,
this is actually one of the biggest
stories behind this whole boom of AI.
The biggest kind of underlying question
is how long would
these things that we spend money on
setting up, like essentially last us?
And
I want to give you like my perspective
from my history with First the
Messenger right?
So, I joined like the Messenger in 2010
when we only had about
100 servers there.
And then I ended up growing that
number of servers to like 2 to 3,000.
Um
at the time, a lot of people were
saying, "Okay, you know, every 3 years
we're going to basically throw away the
old servers and buy new servers."
But
for the Messenger, while I was doing
this to, you know, set up our
infrastructure,
I kind always thought
Okay, 4 years have passed. This server
is still running.
It still has memory and CPUs.
Why throw it away, right? Like
I guess when you do the math, the price
of buying the new server
and you know, the new server is
basically for roughly
Back then, like the price of servers
would almost not change. So, the servers
would always be like around $5,000 each.
These were really
not the Nvidia servers that we have
today that are like crazy expensive.
But, the servers will be roughly 5,000
and
throughout the 10 years I was there,
like that price didn't really change up
or down. It stayed around this number.
But, what you got every year is like
some
better CPUs and some faster memory in
each server.
So, so Anyway, my my long story here is
like at the at IMO
we ended up not throwing away any
machines. We kept them for 5 years and 7
years. We would just move them a little
bit from like high-priority applications
to low-priority applications. So, when
the machines got older
I would just, for example, use them in
our Hadoop clusters to
to data analytics.
>> Right.
>> Because a failure of a machine there
would basically have no effect. While
failure of a machine serving live
customer traffic is
kind of more important.
And so, we I think we did a really good
job there of using the infrastructure
really efficiently.
And
and we were really scrappy. And so, so
the big question here is like how long
would the servers that we buy now be
useful for? And
yes, Nvidia comes up with better and
better GPUs all the time.
But, what we've seen is that
even the oldest
servers we have, the A100s or the H100s
that are now like 2 and 1/2 years old,
are still
pretty fully used. And and even the
prices that we would charge for them has
gone higher.
>> Interesting.
>> Mostly the function of the demand.
>> Mhm.
>> Um
I feel like
we will keep using this infrastructure.
We'll build new ones, but there's so
much demand coming
down the line that
even the oldest ones would still be
pretty used.
>> Okay.
Um
you have a pretty good perspective of
this. So, what is the most overrated
thing in AI infrastructure right now,
you think? And where's, you know, in a
few years we'll be like, "What the
heck?"
>> It's a good question. I think like
I think like right now
we're not asking ourselves like often
the questions like, "Do I really need
all of this? And what part of this do I
really need?"
And so, I try to keep asking this
question all the time like, "This thing
that we bought, are we using it? And how
we use it? Is it
And so,
at the moment, I'll tell you like
what I think and I'm I get a little bit
in trouble like
Nvidia has
generated these reference designs of
like how to build a cluster using their
GPUs and their networking switches and
and all in between.
And
this has been important for training.
I think the designs are mostly
following like what do you really need
to do efficient training.
But,
I feel like inference is quite
different, so it just requires a
different design and there different
questions. Do we need these parts? What
are we going to interconnect them? How
do we build a network that is good for
inference?
And uh
So, that's the part where I think like
we could optimize more the
infrastructure.
Um
Yeah, and the other part is maybe
storage. Like, people seem to I guess
buy certain amount of storage and pay
like quite a bit of premium for like
these fancy storage
providers where I don't know that you
really
how much do you really need that?
Uh depending on what you're building.
What what I like to do is
you set up something and then you have
to look at it and see like, "Okay,
I'm using the GPUs at this level.
Am Am I using the memory? How much of
the memory I'm actually using?" So, not
not too many people can give you like
just a number. Out of all the memory
we've bought,
how much is free right now and how much
is used?
And this is like an important number,
right?
When you buy your next machines, if
you're having so much free memory, then
you should be buying a little less
memory per machine.
And the same questions for the CPUs.
Like, how much of your CPUs are you
actually currently utilizing? Like, what
percent of the cores are idle versus
busy?
And the same thing for the switches and
the other things. And this is what we
did a lot at the messenger. Like, we
basically tracked graphs of There was
four key resources, like memory, CPU,
disks, and networking.
And so, we would
have graphs data center wide of like how
far are we utilizing each of these and
then
it will impact our
next purchases so that we kind of make
sure that
the mix is right.
>> Okay.
Nikola, I just wanted to say thank you
so much for these answers. Uh they are
going to
um
help my audience get even more
interested about AI and sort of the
hardware and the infrastructure that
goes behind all of the the cool little
requests that we talk to to all of our
AIs every day. So thanks for that and I
we appreciate you coming on.
>> Thank you so much. Pleasure.