Playing around with the s2protocol

May 17th, 2013 by Andreas Hofmann

Finally, last week Blizzard has released something official to parse replays properly: s2protocol. Before that, there were a bunch of scripts parsing StarCraft 2 replays more or less successfully. But with every SC2 patch, you had to be afraid, that you’re script isn’t working anymore.
Blizzard’s now saying that they’re supporting every old and every new version of the replays:

s2protocol supports all StarCraft II replay files that were written with retail versions of the game. The current plan is to support all future publicly released versions, including public betas.

The s2protocol is written in python and comes along with a small python script to parse a replay already and print the results as python-objects/dicts, that’s pretty cool for easy and quick testing of it’s abilities.
The great thing is, that it provides you with such a huge amount of data, from the lobby states (slots, etc.), over player settings, battlenet-IDs, regions and for sure all actions done in the match from moving the camera, selecting, moving and building units, chat messages, etc… Ah and for sure the players and the results ;) And muuuuch more!
The horrible thing is: It’s not documented! It… is… NOT… DOCUMENTED!! And this can drive you crazy! I was facing a couple of problems for which I haven’t found any solutions in the web yet (neither temliquid nor day9…) but which I was able to solve. And I want to to tell you about what I have found out until yet:

General info

First I want to tell you what you can ecpect from the different data the script is giving you:

Trackerevents

Everything related to the units/infrastructure of a player. Here you can find events when a unit is completed, has died, was moved or an upgrade is finished.
One thing which took me a while to understand is, what they write at GitHub for tracker events:

NNet.Replay.Tracker.SUnitInitEvent events appear for units under construction. When complete you’ll see a NNet.Replay.Tracker.SUnitDoneEvent with the same unit tag.
NNet.Replay.Tracker.SUnitBornEvent events appear for units that are created fully constructed.

A unit is not only a Probe, a Medivac or an Ultralisk. Also buildings are units! And I was asking myself “Which army-unit is appearing fully constructed? Every unit has a production time, except the MULE”. But you have to think from the replays perspective: For the replay, every army-unit appears fully constructed, because when it’s in the build process, it’s not appearing on the map!
So for army-units, you will only receive a NNet.Replay.Tracker.SUnitBornEvent event, For buildings, you will receive a NNet.Replay.Tracker.SUnitInitEvent event when the build process is initiated and a NNet.Replay.Tracker.SUnitDoneEvent when the building is complete!

A complete list of events can be found here: https://github.com/Blizzard/s2protocol/blob/master/protocol24944.py#L307

Gameevents

This is basically everything and more what counts into the APM of a player. Selecting units, grouping units, moving them, moving the camera, etc.

A complete list of events can be found here: https://github.com/Blizzard/s2protocol/blob/master/protocol24944.py#L204

Details

General match information like the mapname, matchtime, playernames and the match result. See the “Problems & solutions” section for how to get full player info and what to to with the matchtime!

Header

Contains the elapsed gameloops and the client version of the replay.

Initdata

This is the biggest information pool, I think. It contains basically the whole lobby data, including the slots, it’s settings, and many information about the player: his settings, bnet ID, if he’s using an own layout, and so on… But more about this initdata in the “Problems & solutions” section for how to get full player info.

Messageevents

Yeah, all chat messages :) And I think also the ingame messages like “game paused” but not sure yet.

Attributeevents

Haven’t taken a look into that yet, will give you an update on this in the next days.

Problems & solutions

What is this weird value inside m_timeUTC?

If you want to find out the matchtime you think “Ah, there is this m_timeUTC field with a timestamp in it… I just convert this and… WTF!?”. Yes: WTF! This thing doesn’t look anything like a UNIX timestamp! To better understand:
UNIX timestamp: 1368796283
A value inside m_timeUTC: 130132642839470955
After a bit of research a college was able to help me out: It is a Windows NT timesamp :) Which means:

Windows NT time is specified as the number of 100 nanosecond intervals since January 1, 1601. UNIX time is specified as the number of seconds since January 1, 1970. There are 134,774 days (or 11,644,473,600 seconds) between these dates.

Funny, heh? So to get a UNIX timestamp out of it, you have to divide it by 10,000,000 and subtract 11,644,473,600 from it. Now you can use it outside of Windows applications ;)

playerID <=> userID? Two IDs?

Yes, two IDs! A player has two IDs in a replay:

  • A userID => Used in the lobby and to identify the users gameevents
  • A playerID => Used in the playerslist in the details section and to identify the players trackerevents

Every user in the lobby has a userID, for sure. That means every observer, every AI and every player! You can find the userID inside the slotlist which can be found in the initdata section.
A playerID instead is only available for… guess it… players, correct! :) The playerID is currently guessed: For the five test replays I have it works everytime: It is the index of the player inside the playerslist of the details section starting with the 1!

How to get a “complete” player and the userID? And what is this toon?

If you take a look at all these gameevents for example, they all have a m_userId field but in the playerlist inside the data of the details section, there is no userID! That’s why… Man, I don’t know… But I know where you can find it:
Inside the initdata section there are two lists including some information:

  • the whole lobbydata including the slots and
  • the initial data of a player, whatever that means…

The initial player data for example includes the player name, the clantag and other information. But the most useful information is inside the slotlist: It contains the userID! With this userID, you know which events belong to which player, finally \o/ And with that knowledge, you can reference this initial player data, because the index of an element in this list is the userID (this is also guessed, but worked for all my tests)!
But again: How to reference from the slotdata to that player inside the playerlist you got from the details section (for example getting the result of a player)? There’s no userID, like I mentioned before. But I found out, that a replay knows about something called toon. It seems to be a unique identifier through the whole battlenet (If  you know more of that, please let me know! Even, what toon means ;) ). It consists of the players

  • Region
  • Program ID (S2 for StarCraft 2)
  • Realm
  • ID (the battlenet ID)

The format is

1
$playerToon = m_region-m_programId-m_realm-m_id

So for example you get something like this: 2-S2-1-1234567
With this information you can reference to an entry inside the playerlist, because the slot has the field m_toonHandle and the playerlist as a field m_toon which contains the toon in pieces.

What’s a gameloop and how to convert them into seconds?

Every event has the field _gameloop which contains the “time” when it happened in the game.  Converting the gameloop is not necessary to detect the build order, but necessary to display it in a human readable format: in hours/minutes/seconds, because it’s better to say “He build the cybernetics core at second X”!
And wow, that took me a time to find out, how to convert it! After multiple tries of Google  research, I finally found a statement from a blizzard developer (“Rotidecs”. And no, the user “turtles” is not me or somebody else from Turtle Entertainment ;) ):

Note that you can use a wait time of zero to wait for the smallest possible amount of time, which is one game loop (or 1/16 of a second).
(source: http://us.battle.net/sc2/en/forum/topic/7004015250#2

So my guess was valid: 16 gameloops are happening in a second, so just divide the gameloop by 16 and you know, which second it is. Also note, that this are ingame seconds, so it depends on the gamespeed! If you, for example divide the m_elapsedGameLoops of the header section by 16, you will get the matchduration in ingame time! To find out how to convert this ingame time to the real seconds happened, please see this article at Liquipedia: http://wiki.teamliquid.net/starcraft2/Game_Speed

Still unknown

Which events are counting into the APM?

Currently, I try to reconstruct the average APM in a match, which can be seen in the APM tab in the replay. But I still don’t know which events to count in… At the moment I’m counting only these gameevents:

  • ‘NNet.Game.SSelectionDeltaEvent’,
  • ‘NNet.Game.SCmdEvent’,
  • ‘NNet.Game.SControlGroupUpdateEvent’,
  • ‘NNet.Game.SGameUserLeaveEvent’

But I don’t know if this is correct and if I have to count also some trackerevents… Or multiply it by the gamespeed difference, as mentioned here (what makes not much sense): http://wiki.teamliquid.net/starcraft2/APM
As far as I’m able to reconstruct it, I’ll let you know!

 

Sooo, I think that’s it from me at this point! If you have any feedback or also found out something, please feel free to contact me or leave a comment.
I hope this was a bit helpful to start working with the s2protocol!

Greetings,

Andy! 

[UPDATE]

I found out, that a user/player has two different IDs in a replay! I’ve edited the article accordingly.

[UPDATE 2]

I found out, how to convert the gameloop and I’ve added my current problem, the APM. I’ve edited the article accordingly.

Posted in Development, s2protocol, Talk | No Comments »
by

Our Scrum modifications part #1: The story discussions

December 6th, 2012 by Andreas Hofmann

Before I start talking about some nasty Scrum stuff, first let me introduce myself: My name’s Andreas Hofmann, born *86 and working for Turtle Entertainment since January 2011 as a Software Developer. Since April 2012 I’m the ScrumMaster here at Turtle and since August 2012 I’m also the trainer for our “Fachinformatiker Fachrichtung Anwendungsentwicklung” trainees. We’re currently having nine developers (excluding me), including one trainee and two externals working in one team (I think I will write our experiences with multiple teams in a later post).

But I want to tell you something about our own Scrum modifications in this and the next posts to come. You should always remember one of the most important facts about Scrum:

The Scrum process is free to be modified!

If there is something that doesn’t work for your company or if you have a new idea: Do it, change it, improove(!) it! But share it with the world, so everyone knows about your experiences!
So in this post I want to talk about the Story discussions we have established.

What was the idea/the impulse?

Well, in some Sprintplannig 1 and some estimations we figured out, that there can be problems with the stories for the developer side as for the product owner side.
For the development side there was

  • Problems understanding the story itself: What should really be done?
  • Further questions, most in detail: What if and how this and how that?
  • Figured out that a UI is needed for a story and it wasn’t present in the SP1!
  • So: Story can’t be estimated and it’s therefore a 100!

So you can guess the problems for the product owners:

  • Stories are not estimated (=100) so they can’t plan properly!
  • Stories are not committed because of missing requirements, UI for example!
  • And this all leads to a messed up planning for the POs and/or the management!
  • Sometimes a PO needs help from the developers to write a story correctly because he’s missing the required technical knowledge to write the story in the way it can be estimated!

You may say:

  • “Open questions? The SP1 is the place to clearify them!”
  • “Stories are not clear enough? So the POs failed in their work!”"

Maybe! But you have still the missing requirements left and and the missing knowledge! And there are more positive things you will get, if you read further:

The solution

We thought: Hey it would be a great idea to talk about stories before seeing them the first time in the estimation and say: WTF!? So we established the storydiscussions which are taking place when there are new stories written or new facts for an existing story.
The timebox is 1 hour. Allowed members are the team, the PO(s), the ScrumMaster and sometimes special guests like the stakeholder to explian the need in more detail or developers from other companies to work with for a particular story.
Participation of each team member is optional, they are not forced to go there, but it’s in their own interest: If there is a problem, it might be unseen by another developer! Your ScrumMaster job here is to get people there but not to many. We found out that 3-5 teammembers is a good number. And you should mix the devs who are going!

The flow

The PO now presents one story and the discussion/questioning is open. Every dev can ask questions, make reservations that something might not work as intended in the story or maybe it’s totally impossible to do. Or he’s just giving feedback. The PO writes down all questions/answers, feedback and refines the story after the meeting to present a more detailled and clear version to the next estimation meeting! The PO can discuss as many stories he wants but remember: timebox!
Your job as a ScrumMaster is also to watch that it’s not getting out of hands: Sometimes you have very “enthusiastic” team members that try to tell the product owner his job and how he should design the product/story. They can make recommendations or reservations but it’s not their job to design the product!

The outcome

So what do you get?

  • More detailled stories!
  • Less 100-stories in estimation meeting!
  • A team which feels more prepared for a story and knows everything it needs to know about!
  • The team can detect stories where they might have to do some evaluations on them first before they can be comitted! Imagine this would be revealed in SP1, when the story is discussed! That would lead to an horrible SP2 of two days and frustrated devs and POs (and therefore a frustrated SM) or they won’t commit it!
  • All required ressources are present: UI, hardware, software, etc.
  • Overall you will get a more motivated team and POs who are feeling more safe in their work!

For sure, we are not skipping the discussions in SP1, but they are getting smaller! There will be questions again, but you will reduce the suprises to a minimum!

Hopefully that was helpful to you! Please leave some feedback if you have any :) Thank you for reading!!

Tags:
Posted in Development, Scrum | 2 Comments »
by

Beginnings of a Software Developer

December 4th, 2012 by David Rynearson

Greetings Fellow Icicles:

The hyper cool caster is here to do some commentary on the ESL Dev team! Maybe they don’t want me doing this, but let’s give it a try anyhow.

Life at the ESL is pretty fun overall, I started work in November and the group of guys and the team here are just really chill. I have been part of many software teams and this one is by far the best. We’re all young guys in our mid 20s who grew up as gamers and computer nerds. It’s really easy to talk to everyone and collaborate on ideas.

The life of a new developer isn’t easy though, the code base is incredibly verbose and deciding what to implement and when takes a lot of careful planning. My starting role is to learn this code base and become familiar with the system. In order to do this I have to do a lot of testing and paired programming. When I feel confident enough I take a task off of our very large task board and try to develop a new feature for all ESL users. It’s never easy and I have to ask for help, but when I do I have a really good team that will help me understand what needs to be done.

Code on,

David “Icicle” Rynearson

Posted in Development, Electronic Sports League, Talk | 2 Comments »
by

Multiple Doctrine connections

November 30th, 2012 by Christian Eikermann

First of all, a little introducation of myself. My name is Christian Eikermann and I’m 22 years old and working as a Software Developer at Turtle Entertainment for more than 4 years.

Now, lets start with the topic..
It is a little bit hard to implemente mutliple connections with Doctrine in Symfony 1.x. We have spend some hours to find out how it could work. Finally, we found a good solution and wrote down a little tutorial, how you can use mutliple Doctrine connections for models.

Goal:
We want to store specific models in a seperated database and additional to have a decoupled Symfony plugin for it.

1. Bind connections to models

Frist we have to bind some models to a seperated connection, that is quite easy with Doctrine. Just add the “connection” key to your schema.yml in the model, that you want to bind to a seperated connection. The value of this key is the internal database name. For example:

1
2
3
4
5
ModelA:
connection: connection_b
columns:
foo: { type: integer, notnull: true }
bar: { type: integer, notnull: true }

Now you can rebuild your models and Doctrine will automatically adding the connection binding in the base model class file like:

1
2
// Connection Component Binding
Doctrine_Manager::getInstance()->bindComponent('ModelA', 'connection_b');

2. Evaluate database.yml in plugins

The next step is to evaluate the database.yml from your plugin config folder. Because Symfony doesn’t evaluate the database.yml file in each plugin config folder, it evaluates the database.yml from the root config folder only!

To solve this problem we have to evalute the database.yml manually. We need a initialized sfContext and sfDatabaseManager, so we connect to the “context.load_factories” event. The listener should run like the following code:

1
2
3
4
5
$configHandler = new sfDatabaseConfigHandler();
$databases = $configHandler->evaluate(array($this->getRootDir() . '/config/databases.yml'));

$databaseManager = $event->getSubject()->getDatabaseManager();
foreach ($databases as $name => $database) $databaseManager->setDatabase($name, $database);

Note: Doctrine sets the latest added connection always as the default connection! Maybe you have to save the current connection before you add  new database connection with:

1
$currentConnection = Doctrine_Manager::getInstance()->getCurrentConnection()->getName();

And restore it after you set your new database connection with:

1
Doctrine_Manager::getInstance()->setCurrentConnection($currentConnection);

3. Loading models

If you are using now the ModelATable to retrieve some models out of your database, Doctrine uses always the default connection. The problem is, that Symfony is not loading the base model files when you are using the table instance. So the connection binding, as described in step 1, is not done and the model “ModelA” is not bound to the seperated connection.

You can simple autoloading your specific models from the plugin with:

1
2
Doctrine_Core::loadModels($this->configuration->getRootDir() . '/lib/model/doctrine/' . $this->getName());
Doctrine_Core::loadModels($this->getRootDir() . '/lib/model');

The problem is also described in two bug tickets of Symfony and doctrine:
http://www.doctrine-project.org/jira/browse/DC-740
http://trac.symfony-project.org/ticket/7689

Now the table is using the right connection and you can start to work with your models ;)

 

 

Tags: ,
Posted in Development | 1 Comment »
by

Wow, a Devblog!

November 29th, 2012 by Andreas Hofmann

It’s a long time ago we posted something here (actually, it’s the first post for me)! This should have an end! We will continue what our Director IT (Thomas Pöhler) has started: give you more interesting and funny stuff from us, Turtle, the ESL, the world and for sure: cats!

As you may recognized: All the old posts are gone! :( I’m currently trying to restore them all but there is no DB backup, but a web-backup instead… So I have to copy/paste/reformat all the posts again… damn… I’ll try to insert one old post per day! Stay tuned!

Posted in Development, Talk | No Comments »
by

Racks successfully relocated

October 26th, 2012 by Thomas Poehler

After exactly 24h (my alarmclock, which was set to 5 am from day the before, rang again) of work we finished the relocation from 5 racks equipped with over 100 physical servers. Including a BladeCenter with 12 Blades, 20 Supermicro TwinServers, over 50 Webservers, 3 MultiCPU MultiCore Database Servers, lots of different small ones and switching infrastructure with a total of 9 switches. A lot of cabeling was completely renewed / relabled and every server is now easily removable through its rails.

In my next post i will go into detail of planning the relocation. There was lot of thinking and sleepless nights involved and i think it can be interesting for you guys in the same situation. Perhaps theres the one point (yes redundancy cabeling between racks does need an aperture if you have the rackwalls installed :)) you forgot to think about which will save you a lot of nerves.

Posted in datacenter, racks | No Comments »
by

moving racks 2012-10-24

October 19th, 2012 by Thomas Poehler

First preperations:

 

Posted in datacenter, racks | No Comments »
by

retro racks

October 8th, 2012 by Thomas Poehler

While reorganizing some directories on our fileserver i found this. 5 years? … wtf?

 

Posted in datacenter, retro | No Comments »
by

Improving the quality of user stories in a multi-team scrum process

March 3rd, 2011 by Philipp Kölmel

At Turtle Entertainment we are employing scrum for over a year. We have three teams each having one dedicated product owner. By a lot of very valuable help from bor!sgloger the teams are working synchronized since the beginning of 2011. All three teams take part in a combined planning and review meeting. This was a great improvement because now we can proceed very efficiently on the company backlog. But yet we are facing the consequences of forcing teams to work together in a highly efficient process.

Why improve quality?

Team synchronization enforces the teams to work together. Also our product owners have to form a team. This unveiled a bunch of problems to us. These problems existed a long time before but became visible now:

  • The isolation of teams was close to total. They did not talk to each other. So knowledge got stuck at a team for example: How does our testing system work? How do I test most efficiently? – Only one team had the answer and they did not share it.
  • Product owner of team A could not explain a story to team B. And vice versa team B was not able to understand product owner A.
  • The product owners tried to go against the lack of communication by writing more “precise” stories. Eventually the stories were to detailed so they were not negotiable anymore. The stories did not comply with the INVEST model.
  • The new synchronized stories were estimated unrealistically. Comparing the sum of story points to our velocity told us that the whole backlog would be done in one sprint. This was totally unrealistic.
  • Due to the unrealistic estimations we are not able to plan what put our company backlog meeting in jeopardy.

These problems are dangerous to our efficiency they reduced the reliability of our scrum process and last but not least they bear down our teams motivation.

The root cause is easy to identify: There is no flow of information between teams & product owners, teams & teams and product owners & product owners.

We need to fix this fast.

What do we need to do?

Our measures:

  • Transparency – We want more discussion of stories upfront. So we need to start to discuss the stories really really early at the beginning of the writing process. We need to make the writing process a transparent part of the scrum process.
  • Guideline – Next we need a guide on how to discuss a story. What is important? What needs to be discussed next? When is the story ready for the selected backlog?
  • Location – We need a room which encourages discussion and enables creativity.
Discussing User Stories

Scrum teams discuss a user story with the product owner (dressed as Mario?! It’s Carneval :-))

How to do this?

Transparency:

  • Get rid of excel files owned only by one specific product owner. We want all the company backlog in one simple powerpoint file. The product owners can keep their backlogs but they need to pitch their new ideas to our management which then puts them into the company backlog. As off there the story only exists in the powerpoint file and gets removed from the backlog.
  • Product owners start to talk about theme-sized stories with the teams. The product owners are responsible to log the conversation between team and product owner. The scrum masters must push their teams to discuss the stories on a regular, reliable basis.

Guideline:

  • Since we are very inexperienced in discussing user stories we need guidelines to help us. This guideline is a “Definition of done for a user story” giving the answer to the question: “Is this story ready for the selected backlog yet?”. A check list is appended to each story so one can easily see the progress of the conversation. At a glace you can see if this story is doing good or not so you hopefully are encouraged to about the story.
  • We can later introduce a “Level of Done” for the story writing process which helps us with transparency also.

Location:

  • The discussion itself must be made transparent. Having three teams it is not possible to have all members discussing one story at the same time. But all members must have equal chances to know what the other teams or product owners said about a story. So we must go for a asynchron discussion by writing on sticky notes and pin them at a story. Take a look at the photos to see this in action.
  • The room where this discussion happens must not be a scrum space or meeting room. We want a living, open and ongoing discussion and even brainstorms and other creative methods should be applied. This leads to the thought of a cosy, relaxed location with enough space and creativity tools like whiteboards, flipcharts, pens in different colors and colored sticky notes.

Summary

At the end of the day we take a bit of team building and mix it up with a lot of transparency to achive better user stories. Nearly no formalization is needed, except for the check lists and user story done definition.

I think this is a good, noninvasive extension to the scrum process, which will be accepted by the teams and product owners very fast. It will take some time until we have the user stories rewritten and discussed as supposed. The (hopefully positive) effect will be visible next month.

 Files:

Posted in Scrum, Talk | No Comments »
by

Setting up a symfony project with PHPUnit on Hudson

October 6th, 2010 by Franz Stelzer

We are using Hudson now for several months as Continous Integration System. This short article describes how we have configured a hudson project for a symfony 1.4 project. I am assuming that the reader is already used to Hudson and knows how a normal project has to be configured.

PHPUnit is the test framework of our choice (surprise, surprise) and we are using the sfPHPUnit2Plugin for all our projects. If you do not now this plugin you may first read another post where the usage and features are described in detail.

All requirements in short:

  • Hudson has to be installed
  • Hudson plugin xUnit Plugin has to be installed
  • the symfony project needs the sfPHPUnit2Plugin
  • PHPUnit has to be installed on your test server

Ok, here the configuration steps of the hudson project:

1. Configure your project

Configure standard settings for a hudson project like source-code management settings or email notifications. Please check the official docs if you do not know how to handle this.

2.  Add a shell build step

Building a symfony project in a test environment is pretty easy. With the help of some shell commands the project is completely configured and ready for testing. Those shell commands may be entered in the build step section of the hudson project. Defining the correct commands is the main part during the configuration process.

Our configuration looks like this:

1
2
3
4
5
6
cd $WORKSPACE/trunk
sh _deployment/install_test.sh
php symfony cc
php symfony phpunit:test-all --configuration --options="--log-junit=build/testresult_$BUILD_NUMBER.xml"
cd build
ln -s -f testresult_$BUILD_NUMBER.xml currentTestResult.xml
  1. Jump in the project root of the project
  2. Install the project on the test server with the help of a internal shell script. This step includes for example the generation of the databases.yml.
  3. Clear the symfony cache (always a good choice)
  4. Run all PHPUnit tests including unit and functional tests. The test result is written in a jUnit compatible logfile (needed for the xUnit Plugin).
  5. Jump in the build directory, which is internally used by Hudson
  6. Symlink the latest testresult

3. Configure Post-Build-Action

After the xUnit Plugin is installed correctly, an additional PHPUnit Pattern field should be displayed in the post build action section. In this field has to be entered:

1
trunk/build/currentTestResult.xml

The options “Fail the build if test results were not updated this run” and “Delete temporary JUnit file” should be both checked.

The xUnit Plugin takes the currentTestResult.xml file, which was previously created with the help of the sfPHPUnit2Plugin and analyzes it. When everything works fine, you should be able to review the created test reports.

Here some screenshots how this result could look like.

Build history:

Build history

Trend graph of the test results:

Trend graph

Tags: , , ,
Posted in symfony | No Comments »
by

« Older Entries