Post news Report RSS Campaign data handling

(Warning: Nerdy contents ahead.) If you haven’t noticed, then I have to tell you that I am focusing a lot on data structures, data handling and data in general. The reason why, is that we have some needs for advanced data structures in order to handle a massive amount of data. But another reason is that we are using an unusual approach to dealing with data client side.

Posted by on

By Reto.Chrestangen, Flash Programmer

(Warning: Nerdy contents ahead.)

If you haven’t noticed, then I have to tell you that I am focusing a lot on data structures, data handling and data in general. The reason why, is that we have some needs for advanced data structures in order to handle a massive amount of data. But another reason is that we are using an unusual approach to dealing with data client side.

database_diagram

Usual you do a lot of data processing on the server side. For instance if I needed to get hold of all the Assault Teams that a user has control over, I would ask the server for this and it figures out which ones I need. This is neat and simple for the client side programming, but on the other hand it uses the servers processing power to do this. This is why a lot of people have used a lot of time building databases and server optimization, so that all this is handled with as much ease as possible. Server programmers also have to keep this in mind when doing expensive server operations such as ‘joins’. As we are creating a MMO-like game there will be a lot of clients (hopefully) asking our servers for information. Therefore we want the servers to do the least amount of work as possible, to keep them snappy. In order to achieve that we have almost no server operations (only login). Instead all data in the database is sent to the client and we leave it to the client to do the data handling.

client_database

Needles to say, this leaves a lot of work for the client. All data is received and then the database structure is replicated. In order to do joins and fast lookups we also create maps (with keys). This structure has been revised and is now being implemented. Quite soon you’ll experience a smooth and fast client side database.

However this also creates some new problem. A huge amount of data is sent, and most of it at the entry of the campaign. That’s why we’re cashing all static data, so the servers don’t have to fish it out of the database and the users don’t have to download it each time. There are still lots of static data being sent and we have not solved this yet, but are thinking of pulling out data from the database each hour and cashing this – and then only sending the changes from that instance to where we are now.

Sign up for an Alpha-key here!

(Heroes & Generals blog link)

Post comment Comments
PierreOfTheFrench
PierreOfTheFrench

Interesting stuff, I always liked the idea of caching time based deltas of the database image, then, when people connect you only have to check their 'sequence number' and apply the deltas since then (kind of like patching). Not sure how well it would work but indexing based on sequence values would make it fast, and it would not have to send updates on every entity.

Keep up the great work guys! Can't wait for an alpha key :)

Reply Good karma Bad karma+1 vote
Baryonyx
Baryonyx

I really hope it works.

Reply Good karma Bad karma+1 vote
cucu
cucu

Although I don't know much about the details, here's a piece of advice. The problem with giving out data manipulation tasks to the client is cheating. If the client is anything more than a GUI (terminal) some players will find a way to hack it which is not so hard as they can observe the communication between the client running on their computer and the server. And in multiplayer and mmo games, it's not really acceptable.

If on the other hand all the operations are done on the server, the game can be only hacked if the server is hacked. And that is way more difficult (and serious as well). There are of course two problems with this:
- lagging: to counter this the client should forecast what's going to happen and synchronize these forecasts with the server (where the real in-game reality takes place)
- performance issues: the core program needs optimization. As I've said I don't know much about this project, so just one experience I had with my project: MySQL MEMORY tables are a great way to enhance performance as I/O is one of the narrowest bottlenecks. As for joins: use the appropriate indices, this will solve 90% of your problems.

Good luck with Heroes & Generals! :-)

Reply Good karma Bad karma+1 vote
PierreOfTheFrench
PierreOfTheFrench

I don't believe this article includes anything secretive, the flow diagram seems to show more situation report based objects (like locations of battlefields/factories).

Since they said it was mostly static, it won't be data manipulation as much as data storage and cache accessing that will need to be updated when information is stale.

"- lagging: to counter this the client should forecast" I don't think network interpolation applies here either, I don't think this type of data needs sub-second response. Unless I missed something too! :)

Reply Good karma Bad karma+1 vote
cucu
cucu

"For instance if I needed to get hold of all the Assault Teams that a user has control over, I would ask the server for this and it figures out which ones I need. This is neat and simple for the client side programming, but on the other hand it uses the servers processing power to do this." "Instead all data in the database is sent to the client and we leave it to the client to do the data handling."

In my reading this means:
1. The server sends all the units and the client selects those I can control. So if I can hack the client (or just the client-server) communication, I can control units I should not be able to.
2. The server sends all info and the client filters those I might see (cf fog of war, spying). So if I can hack the client, I can see everything.

This applies to static data as well (static map vs fog of war).

Lagging: the homepage says "The Action game is a modern multiplayer First Person Shooter game". If you have a tank on a field and I have another one there, the position of the two tanks must be computed on the server (otherwise I just send manipulated data about the position of my tank) and be sent to both clients all the time. Without clientside forecasting this will lag.

If all data like that is handled on the server, then of course you are right. But the article suggests otherwise: "In order to achieve that we have almost no server operations (only login). Instead all data in the database is sent to the client and we leave it to the client to do the data handling."

Reply Good karma Bad karma+1 vote
PierreOfTheFrench
PierreOfTheFrench

Yeah I think we are interpreting the article in two different perspectives :) I see what you are saying with the first part, I agree, ALL information including non-privileged should NOT be sent for the client to self-regulate. It was my understanding that the data in question was just extremely large amounts of situation report data that was static and could be seen by anyone.

When they say server should just handle logins, I was thinking they were talking about the Campaign server itself (since from what I know about the game, each battle is instanced separately (so collision/physics would not apply to this server/article and instead to a separate set of servers)

Reply Good karma Bad karma+1 vote
cucu
cucu

Btw if you check the game's blog ( Heroesandgenerals.com ) I'm not the only one who has concerns about this issue.

Reply Good karma Bad karma+1 vote
Christen
Christen

I can see there is a mix up between the action and the strategy part of the game. These run on different servers, but uses some of the same data. I sorry that I did not write the blog less ambiguous.

@cucu
I have answered most of your question here: (I'm not allowed to post urls, so you have to piece it together.) heroesandgenerals.com /community/5137/campaign-data-handling
In reguard to:
1. "...I can control units I should not be able to."
- Yes you can client side, only on your computer. If you try to send it to the server. It will reply, "no can do"!
2. "...(cf fog of war, spying)..."
- As we have no fog of war at the moment this is not a problem. If we implement fog of war on the campaign part we will have to look into it. (If you want to spy, you could just create an account on the opposite side..?)

@PierreOfTheFrench
Thanks for your interpreting, :)
We will look into the "time based deltas" (thanks for the input). I guess it is kind of the same idea that we have in cashing every hour.

Reply Good karma Bad karma+1 vote
cucu
cucu

Thanks for the answer. Just one minor note: by spying I meant getting any kind of info that only another player can see. Like the health point or ammo of an enemy unit. I suppose I cannot create accounts to see everything (right?), so if the client does receive everything (and filters it after that), I can see more than I should.

Reply Good karma Bad karma+1 vote
Post a comment

Your comment will be anonymous unless you join the community. Or sign in with your social account: