Couchbase as memcached replacement – find out expiration time of your keys

December 11th, 2013 by Thomas Poehler

www.esl.eu is making heavy us of the memcache daemon as an indispensable caching layer. It is very simple and rockstable. Our servers have an uptime of currently 412 days. And thats only because we had to move our racks to another datacenter. But theres some big downsides of memcached. It does not scale, has very few statistics to determine performance problems and no automatic failover mechanics.

So about a year ago for me “nosql” was the new buzzword. Pretty much reminded me of memcached in the first place. Some research brought up couchbase with a 100% compatible memcached api. So we started to try that out. Felt pretty stable and the installation was painless. They provide a debian package which was puppetized in no time. A few month in development stage we made the decision to give it a try in production.

Now we are transitioning our memcached cluster to a couchbase cluster. Our production cluster is starting with 5 nodes with each 16 GB RAM and SSD backed. In this transition phase we are letting our memcached class write both stores; the still active memcached cluster and additionally in our new couchbase cluster. We are gathering performance statistics and its the first time we are able to see really lots of stuff which memcache was not providing. For example the total number of items in the cache. (correction total numbers are actually listed) but to make it short: memcache does not provide stats like these

The big difference between memcached and couchbase is its purpose. memcached is meant a caching layer, couchbase is a key / value database. Where for memcache it is ok to evict keys, a database should never lose data. So thats why couchbase needs a disk backend to persist data. This on the other hand makes you care about TTL times on your keys. If you have no TTL set, couchbase will store them forever and will end up being blocked by no space left on device.

Heres a little couchbase view to find out how many and which keys have no expiration set in your couchbase bucket.

1
2
3
4
5
function (doc, meta) {
if (meta.expiration == 0) {
emit(meta.id, meta.expiration);
}
}

if you want the total count of the items just add the built-in reduce function called _count. Make sure to apply the view on the full data set to get real numbers.

I will share more expierence with couchbase over time.

“My life for Aiur!” – Open source SC2 replay analyzer

June 20th, 2013 by Andreas Hofmann

After we released some docs for the s2protocol by Blizzard Entertainment (https://github.com/Blizzard/s2protocol), I’m sure you know, that we are building something and here it is:

Aiur, our replay parser and analyzer based on the s2protocol, 100% free, Python powered and: Glutenfree! Hosted on GitHub: https://github.com/TurtleEntertainment/aiur

What’s so special about it? Well, compared to sc2gears, sc2reader, etc. nothing, because they are more powerful, but Aiur is using the s2protocol and therefore is more safe in reading the replays instead of guessing. Please don’t understand me wrong: I don’t want to talk bad about sc2gears, sc2reader, etc., for gods sake, no! These are great, great tools/projects but if we want to analyze replays for the ESL, we need something different, so we developed Aiur! :)
And we said: Hey, let’s make this public, let the people out there feel a bit of the ESL and let them develop even better things based on what we have started! :)

For sure, we will improve this project over the time and give it more and more features. But let’s start small! :)

So: Feel free to use, fork and/or develop with it! glhf!!

 

Playing around with the s2protocol

May 17th, 2013 by Andreas Hofmann

Finally, last week Blizzard has released something official to parse replays properly: s2protocol. Before that, there were a bunch of scripts parsing StarCraft 2 replays more or less successfully. But with every SC2 patch, you had to be afraid, that you’re script isn’t working anymore.
Blizzard’s now saying that they’re supporting every old and every new version of the replays:

s2protocol supports all StarCraft II replay files that were written with retail versions of the game. The current plan is to support all future publicly released versions, including public betas.

The s2protocol is written in python and comes along with a small python script to parse a replay already and print the results as python-objects/dicts, that’s pretty cool for easy and quick testing of it’s abilities.
The great thing is, that it provides you with such a huge amount of data, from the lobby states (slots, etc.), over player settings, battlenet-IDs, regions and for sure all actions done in the match from moving the camera, selecting, moving and building units, chat messages, etc… Ah and for sure the players and the results ;) And muuuuch more!
The horrible thing is: It’s not documented! It… is… NOT… DOCUMENTED!! And this can drive you crazy! I was facing a couple of problems for which I haven’t found any solutions in the web yet (neither temliquid nor day9…) but which I was able to solve. And I want to to tell you about what I have found out until yet:

General info

First I want to tell you what you can ecpect from the different data the script is giving you:

Trackerevents

Everything related to the units/infrastructure of a player. Here you can find events when a unit is completed, has died, was moved or an upgrade is finished.
One thing which took me a while to understand is, what they write at GitHub for tracker events:

NNet.Replay.Tracker.SUnitInitEvent events appear for units under construction. When complete you’ll see a NNet.Replay.Tracker.SUnitDoneEvent with the same unit tag.
NNet.Replay.Tracker.SUnitBornEvent events appear for units that are created fully constructed.

A unit is not only a Probe, a Medivac or an Ultralisk. Also buildings are units! And I was asking myself “Which army-unit is appearing fully constructed? Every unit has a production time, except the MULE”. But you have to think from the replays perspective: For the replay, every army-unit appears fully constructed, because when it’s in the build process, it’s not appearing on the map!
So for army-units, you will only receive a NNet.Replay.Tracker.SUnitBornEvent event, For buildings, you will receive a NNet.Replay.Tracker.SUnitInitEvent event when the build process is initiated and a NNet.Replay.Tracker.SUnitDoneEvent when the building is complete!

A complete list of events can be found here: https://github.com/Blizzard/s2protocol/blob/master/protocol24944.py#L307

Gameevents

This is basically everything and more what counts into the APM of a player. Selecting units, grouping units, moving them, moving the camera, etc.

A complete list of events can be found here: https://github.com/Blizzard/s2protocol/blob/master/protocol24944.py#L204

Details

General match information like the mapname, matchtime, playernames and the match result. See the “Problems & solutions” section for how to get full player info and what to to with the matchtime!

Header

Contains the elapsed gameloops and the client version of the replay.

Initdata

This is the biggest information pool, I think. It contains basically the whole lobby data, including the slots, it’s settings, and many information about the player: his settings, bnet ID, if he’s using an own layout, and so on… But more about this initdata in the “Problems & solutions” section for how to get full player info.

Messageevents

Yeah, all chat messages :) And I think also the ingame messages like “game paused” but not sure yet.

Attributeevents

Haven’t taken a look into that yet, will give you an update on this in the next days.

Problems & solutions

What is this weird value inside m_timeUTC?

If you want to find out the matchtime you think “Ah, there is this m_timeUTC field with a timestamp in it… I just convert this and… WTF!?”. Yes: WTF! This thing doesn’t look anything like a UNIX timestamp! To better understand:
UNIX timestamp: 1368796283
A value inside m_timeUTC: 130132642839470955
After a bit of research a college was able to help me out: It is a Windows NT timesamp :) Which means:

Windows NT time is specified as the number of 100 nanosecond intervals since January 1st, 1601. UNIX time is specified as the number of seconds since January 1st, 1970. There are 134,774 days (or 11,644,473,600 seconds) between these dates.

Funny, heh? So to get a UNIX timestamp out of it, you have to divide it by 10,000,000 and subtract 11,644,473,600 from it. Now you can use it outside of Windows applications ;)

And what to do with the m_timeLocalOffset field?

Yeah, that’s also a bit weird, but if you know how to handle the m_timeUTC field above, it’s also “easy” to decode this field:
This is, as expected, the UTC timezone offset for the matchtime in hours, but (as the m_timeUTC field) it’s stored in 100 nanoseconds! So to convert it back to hours, simply divide it by 36000000000 (= 60*60*10^7 (10^7, because we need the 100 nanoseconds!))! That’s it! :)

playerID <=> userID? Two IDs?

Yes, two IDs! A player has two IDs in a replay:

  • A userID => Used in the lobby and to identify the users gameevents
  • A playerID => Used in the playerslist in the details section and to identify the players trackerevents

Every user in the lobby has a userID, for sure. That means every observer, every AI and every player! You can find the userID inside the slotlist which can be found in the initdata section.
A playerID instead is only available for… guess it… players, correct! :) The playerID is currently guessed: For the five test replays I have it works everytime: It is the index of the player inside the playerslist of the details section starting with the 1!

How to get a “complete” player and the userID? And what is this toon?

If you take a look at all these gameevents for example, they all have a m_userId field but in the playerlist inside the data of the details section, there is no userID! That’s why… Man, I don’t know… But I know where you can find it:
Inside the initdata section there are two lists including some information:

  • the whole lobbydata including the slots and
  • the initial data of a player, whatever that means…

The initial player data for example includes the player name, the clantag and other information. But the most useful information is inside the slotlist: It contains the userID! With this userID, you know which events belong to which player, finally \o/ And with that knowledge, you can reference this initial player data, because the index of an element in this list is the userID (this is also guessed, but worked for all my tests)!
But again: How to reference from the slotdata to that player inside the playerlist you got from the details section (for example getting the result of a player)? There’s no userID, like I mentioned before. But I found out, that a replay knows about something called toon. It seems to be a unique identifier through the whole battlenet (If  you know more of that, please let me know! Even, what toon means ;) ). It consists of the players

  • Region
  • Program ID (S2 for StarCraft 2)
  • Realm
  • ID (the battlenet ID)

The format is

1
$playerToon = m_region-m_programId-m_realm-m_id

So for example you get something like this: 2-S2-1-1234567
With this information you can reference to an entry inside the playerlist, because the slot has the field m_toonHandle and the playerlist as a field m_toon which contains the toon in pieces.

What’s a gameloop and how to convert them into seconds?

Every event has the field _gameloop which contains the “time” when it happened in the game.  Converting the gameloop is not necessary to detect the build order, but necessary to display it in a human readable format: in hours/minutes/seconds, because it’s better to say “He build the cybernetics core at second X”!
And wow, that took me a time to find out, how to convert it! After multiple tries of Google  research, I finally found a statement from a blizzard developer (“Rotidecs”. And no, the user “turtles” is not me or somebody else from Turtle Entertainment ;) ):

Note that you can use a wait time of zero to wait for the smallest possible amount of time, which is one game loop (or 1/16 of a second).
(source: http://us.battle.net/sc2/en/forum/topic/7004015250#2

So my guess was valid: 16 gameloops are happening in a second, so just divide the gameloop by 16 and you know, which second it is. Also note, that this are ingame seconds, so it depends on the gamespeed! If you, for example divide the m_elapsedGameLoops of the header section by 16, you will get the matchduration in ingame time! To find out how to convert this ingame time to the real seconds happened, please see this article at Liquipedia: http://wiki.teamliquid.net/starcraft2/Game_Speed

Still unknown

Which events are counting into the APM?

Currently, I try to reconstruct the average APM in a match, which can be seen in the APM tab in the replay. But I still don’t know which events to count in… At the moment I’m counting only these gameevents:

  • ‘NNet.Game.SSelectionDeltaEvent’,
  • ‘NNet.Game.SCmdEvent’,
  • ‘NNet.Game.SControlGroupUpdateEvent’,
  • ‘NNet.Game.SGameUserLeaveEvent’

But I don’t know if this is correct and if I have to count also some trackerevents… Or multiply it by the gamespeed difference, as mentioned here (what makes not much sense): http://wiki.teamliquid.net/starcraft2/APM
As far as I’m able to reconstruct it, I’ll let you know!

 

Sooo, I think that’s it from me at this point! If you have any feedback or also found out something, please feel free to contact me or leave a comment.
I hope this was a bit helpful to start working with the s2protocol!

Greetings,

Andy! 

[UPDATE]

I found out, that a user/player has two different IDs in a replay! I’ve edited the article accordingly.

[UPDATE 2]

I found out, how to convert the gameloop and I’ve added my current problem, the APM. I’ve edited the article accordingly.

[UPDATE 3]

And more found out: Now I can also read the value inside the m_timeLocalOffset of the details section. See above!

Our Scrum modifications part #1: The story discussions

December 6th, 2012 by Andreas Hofmann

Before I start talking about some nasty Scrum stuff, first let me introduce myself: My name’s Andreas Hofmann, born *86 and working for Turtle Entertainment since January 2011 as a Software Developer. Since April 2012 I’m the ScrumMaster here at Turtle and since August 2012 I’m also the trainer for our “Fachinformatiker Fachrichtung Anwendungsentwicklung” trainees. We’re currently having nine developers (excluding me), including one trainee and two externals working in one team (I think I will write our experiences with multiple teams in a later post).

But I want to tell you something about our own Scrum modifications in this and the next posts to come. You should always remember one of the most important facts about Scrum:

The Scrum process is free to be modified!

If there is something that doesn’t work for your company or if you have a new idea: Do it, change it, improove(!) it! But share it with the world, so everyone knows about your experiences!
So in this post I want to talk about the Story discussions we have established.

What was the idea/the impulse?

Well, in some Sprintplannig 1 and some estimations we figured out, that there can be problems with the stories for the developer side as for the product owner side.
For the development side there was

  • Problems understanding the story itself: What should really be done?
  • Further questions, most in detail: What if and how this and how that?
  • Figured out that a UI is needed for a story and it wasn’t present in the SP1!
  • So: Story can’t be estimated and it’s therefore a 100!

So you can guess the problems for the product owners:

  • Stories are not estimated (=100) so they can’t plan properly!
  • Stories are not committed because of missing requirements, UI for example!
  • And this all leads to a messed up planning for the POs and/or the management!
  • Sometimes a PO needs help from the developers to write a story correctly because he’s missing the required technical knowledge to write the story in the way it can be estimated!

You may say:

  • “Open questions? The SP1 is the place to clearify them!”
  • “Stories are not clear enough? So the POs failed in their work!”"

Maybe! But you have still the missing requirements left and and the missing knowledge! And there are more positive things you will get, if you read further:

The solution

We thought: Hey it would be a great idea to talk about stories before seeing them the first time in the estimation and say: WTF!? So we established the storydiscussions which are taking place when there are new stories written or new facts for an existing story.
The timebox is 1 hour. Allowed members are the team, the PO(s), the ScrumMaster and sometimes special guests like the stakeholder to explian the need in more detail or developers from other companies to work with for a particular story.
Participation of each team member is optional, they are not forced to go there, but it’s in their own interest: If there is a problem, it might be unseen by another developer! Your ScrumMaster job here is to get people there but not to many. We found out that 3-5 teammembers is a good number. And you should mix the devs who are going!

The flow

The PO now presents one story and the discussion/questioning is open. Every dev can ask questions, make reservations that something might not work as intended in the story or maybe it’s totally impossible to do. Or he’s just giving feedback. The PO writes down all questions/answers, feedback and refines the story after the meeting to present a more detailled and clear version to the next estimation meeting! The PO can discuss as many stories he wants but remember: timebox!
Your job as a ScrumMaster is also to watch that it’s not getting out of hands: Sometimes you have very “enthusiastic” team members that try to tell the product owner his job and how he should design the product/story. They can make recommendations or reservations but it’s not their job to design the product!

The outcome

So what do you get?

  • More detailled stories!
  • Less 100-stories in estimation meeting!
  • A team which feels more prepared for a story and knows everything it needs to know about!
  • The team can detect stories where they might have to do some evaluations on them first before they can be comitted! Imagine this would be revealed in SP1, when the story is discussed! That would lead to an horrible SP2 of two days and frustrated devs and POs (and therefore a frustrated SM) or they won’t commit it!
  • All required ressources are present: UI, hardware, software, etc.
  • Overall you will get a more motivated team and POs who are feeling more safe in their work!

For sure, we are not skipping the discussions in SP1, but they are getting smaller! There will be questions again, but you will reduce the suprises to a minimum!

Hopefully that was helpful to you! Please leave some feedback if you have any :) Thank you for reading!!

Beginnings of a Software Developer

December 4th, 2012 by David Rynearson

Greetings Fellow Icicles:

The hyper cool caster is here to do some commentary on the ESL Dev team! Maybe they don’t want me doing this, but let’s give it a try anyhow.

Life at the ESL is pretty fun overall, I started work in November and the group of guys and the team here are just really chill. I have been part of many software teams and this one is by far the best. We’re all young guys in our mid 20s who grew up as gamers and computer nerds. It’s really easy to talk to everyone and collaborate on ideas.

The life of a new developer isn’t easy though, the code base is incredibly verbose and deciding what to implement and when takes a lot of careful planning. My starting role is to learn this code base and become familiar with the system. In order to do this I have to do a lot of testing and paired programming. When I feel confident enough I take a task off of our very large task board and try to develop a new feature for all ESL users. It’s never easy and I have to ask for help, but when I do I have a really good team that will help me understand what needs to be done.

Code on,

David “Icicle” Rynearson

Multiple Doctrine connections

November 30th, 2012 by Christian Eikermann

First of all, a little introducation of myself. My name is Christian Eikermann and I’m 22 years old and working as a Software Developer at Turtle Entertainment for more than 4 years.

Now, lets start with the topic..
It is a little bit hard to implemente mutliple connections with Doctrine in Symfony 1.x. We have spend some hours to find out how it could work. Finally, we found a good solution and wrote down a little tutorial, how you can use mutliple Doctrine connections for models.

Goal:
We want to store specific models in a seperated database and additional to have a decoupled Symfony plugin for it.

1. Bind connections to models

Frist we have to bind some models to a seperated connection, that is quite easy with Doctrine. Just add the “connection” key to your schema.yml in the model, that you want to bind to a seperated connection. The value of this key is the internal database name. For example:

1
2
3
4
5
ModelA:
connection: connection_b
columns:
foo: { type: integer, notnull: true }
bar: { type: integer, notnull: true }

Now you can rebuild your models and Doctrine will automatically adding the connection binding in the base model class file like:

1
2
// Connection Component Binding
Doctrine_Manager::getInstance()->bindComponent('ModelA', 'connection_b');

2. Evaluate database.yml in plugins

The next step is to evaluate the database.yml from your plugin config folder. Because Symfony doesn’t evaluate the database.yml file in each plugin config folder, it evaluates the database.yml from the root config folder only!

To solve this problem we have to evalute the database.yml manually. We need a initialized sfContext and sfDatabaseManager, so we connect to the “context.load_factories” event. The listener should run like the following code:

1
2
3
4
5
$configHandler = new sfDatabaseConfigHandler();
$databases = $configHandler->evaluate(array($this->getRootDir() . '/config/databases.yml'));

$databaseManager = $event->getSubject()->getDatabaseManager();
foreach ($databases as $name => $database) $databaseManager->setDatabase($name, $database);

Note: Doctrine sets the latest added connection always as the default connection! Maybe you have to save the current connection before you add  new database connection with:

1
$currentConnection = Doctrine_Manager::getInstance()->getCurrentConnection()->getName();

And restore it after you set your new database connection with:

1
Doctrine_Manager::getInstance()->setCurrentConnection($currentConnection);

3. Loading models

If you are using now the ModelATable to retrieve some models out of your database, Doctrine uses always the default connection. The problem is, that Symfony is not loading the base model files when you are using the table instance. So the connection binding, as described in step 1, is not done and the model “ModelA” is not bound to the seperated connection.

You can simple autoloading your specific models from the plugin with:

1
2
Doctrine_Core::loadModels($this->configuration->getRootDir() . '/lib/model/doctrine/' . $this->getName());
Doctrine_Core::loadModels($this->getRootDir() . '/lib/model');

The problem is also described in two bug tickets of Symfony and doctrine:
http://www.doctrine-project.org/jira/browse/DC-740
http://trac.symfony-project.org/ticket/7689

Now the table is using the right connection and you can start to work with your models ;)

 

 

Wow, a Devblog!

November 29th, 2012 by Andreas Hofmann

It’s a long time ago we posted something here (actually, it’s the first post for me)! This should have an end! We will continue what our Director IT (Thomas Pöhler) has started: give you more interesting and funny stuff from us, Turtle, the ESL, the world and for sure: cats!

As you may recognized: All the old posts are gone! :( I’m currently trying to restore them all but there is no DB backup, but a web-backup instead… So I have to copy/paste/reformat all the posts again… damn… I’ll try to insert one old post per day! Stay tuned!

Racks successfully relocated

October 26th, 2012 by Thomas Poehler

After exactly 24h (my alarmclock, which was set to 5 am from day the before, rang again) of work we finished the relocation from 5 racks equipped with over 100 physical servers. Including a BladeCenter with 12 Blades, 20 Supermicro TwinServers, over 50 Webservers, 3 MultiCPU MultiCore Database Servers, lots of different small ones and switching infrastructure with a total of 9 switches. A lot of cabeling was completely renewed / relabled and every server is now easily removable through its rails.

In my next post i will go into detail of planning the relocation. There was lot of thinking and sleepless nights involved and i think it can be interesting for you guys in the same situation. Perhaps theres the one point (yes redundancy cabeling between racks does need an aperture if you have the rackwalls installed :)) you forgot to think about which will save you a lot of nerves.

moving racks 2012-10-24

October 19th, 2012 by Thomas Poehler

First preperations:

 

retro racks

October 8th, 2012 by Thomas Poehler

While reorganizing some directories on our fileserver i found this. 5 years? … wtf?