DevChat Log - November 7th, 2009

[odwill]

<@bobius> Please ask all questions in #devchat-questions
<+learningdisease> Please ask all questions in #devchat-questions
<@Serpentkaa> Hello Everyone. Welcome to DevChat. Let's get started. First off, let me tell you how this will proceed.
<@Serpentkaa> Please ask your questions in #devchat-questions. We will paste the question here in this channel. We will answwer as many as we can.
<@Serpentkaa> We have Bobius, cRush and Anakis here today.
<@bobius>

<Austen> When the OR is done, will this ham glitch (the one where you regenerate FULL HAM via the use of a food) be fixed so pvp is no longer full of people using food to regain their Ham?
<@cRush> Yes
<@bobius> OR will address a lot of the outstanding bugs. It will hopefully be the final iteration of the codebase that will get us to 1.0

<Coolcat> Question: Even if I know the answer, I'll ask this one for the benefit of everyone: Mozilla Firefox reports the SWGemu forums as an attack site, are the forums trying to hack my computer or is Firefox wrong?
<@bobius> We're looking into the matter now. Basically what we think happened was a flash ad from google's adsense contained malicious code that redirected some users to bad sites. Google then flagged our site. Ironically it was google's inability to properly audit it's 3rd party code that got us into this mess.
<@bobius> We've filed a ticket with Google and hopefully the warnings will go away soon. We've checked the site's code in no data was lost and it currently does not contain links to malicious websites

[Nemesidian] Will there be another wipe of all characters?
<@bobius> Short answer: Yes.
<@bobius> This is a Test Center. You can expect wipes at any time. However, we do realize that players put a lot of time an effort into their characters. We will never wipe without a reason.

<@bobius> Welcome Oru, to the Developer Chat
<&Oru> Hi everyone, sorry for being late

<Chunta> any eta on player structures?
<Austen> With the implementation of structures, will we see massive jumps in our ping just as we did in live?
<@cRush> Buildings by themselves generate no more lag than any object in game as far as the network is concerned. However, they can produce graphical lag.
<@cRush> Upon entering a building, however, all items are then sent to your computer, producing a small, but noticeable lag spike.
<@bobius> It has always been our policy not to comment on ETA. We are focusing on Restructuring our code base at the moment. After that we will focus on adding additional features.
<@cRush> Buildings will not be introduced before the Object Restructure, however.
<@cRush> As a general rule, the TC will stay in its current state until the Object Restructure is ready for testing. There may be some small fixes occasionally, but no features.

<+Ellayn> > [Likto] SInce just before the last wipe, I've been here with Nova test. The whole time I've heard a lot about hackers and speed hacking and suchagainst our game. What progresses have bene made to eliminate those troubles and do I need to set up a firewal comp here or are the devs able to protect us?
<@bobius> First of all, let me put your mind at ease. The "hackers" in no way can affect your home computer.

<Likto> Why is it a problem to not have the maximum player cap set to higher so we get less lag? willwe be getting a higher player cap number when we go live?
<@bobius> There have been some pretty clever 3rd party tools going around that allowed players to change their speed or warp to different locations on the map.
<@bobius> We are in the process of making those tools ineffective.
<@bobius> However, this is why you're currently rubberbanding or getting "stuck in a box"
<@bobius> we are tweaking the tools every day to try to prevent this from happening, but it is not an easy problem to fix.
<@bobius> As far as the second question
<&Oru> As long as we are having a single node server setup, we cannot raise the capacity of the server. We are working on multi-node version where the workload will be distributed among several physical servers and then we can support a whole lot more of users.
<@bobius> The OR will also be much more memory efficient and will have automatic memory management and cleanup that is not currently running on TC. This will allow more players per server as well

<kamuirsx> Will you guys put the correct harvesting resources rates on creatures that have been overlooked like Woolymanders on the TC while focusing on the whatever you're doing?
<@cRush> The scope and mission statement of this project entails this very type of thing. It is always our focus to mimic with 100% accuracy the state of Pre-CU at patch 14.1. It will take some tweaking to get there, and there are more important hurdles that must be jumped first.
<@cRush> Currently, however, TC will not be adjusted until the Object Restructure is implemented.

<Kyrm347> Will the OR implementation allow us to once again test the entire Inventory and Crafting systems as well as Droids and Pets, or will parts of these features be implemented in stages after OR?
<@bobius> Eventually, yes. We have no yet decided at which stage we will make the OR public.
<@bobius> All systems will have to be retested to insure we did not mess anything up during the conversion.
<@bobius> To clarify what I just said: The OR codebase is public and can be accessed via our svn as usual. It is not yet publicly available to play on the TC


<@cRush> We are going to take a brief break from answering questions to address what exactly the Object Restructure (OR) is.
<@bobius> Please give us a few moments to gather our thoughts to explain it as efficiently as possible
<@bobius> As we progressed further and further with the project, we learned quite a bit.
<@bobius> The more we learned about how the client worked, the more we realized we had a few things backwards.
<@bobius> TA and Oru started rewriting parts of the engine to fix these problems.
<@bobius> That engine is now finished.
<@bobius> We developers are in the process of reevaluating all the currently implemented systems to make sure they are designed with efficiency, security, and scaleability in mind.
<@bobius> Right now we can support over 1000 current users.
<@bobius> We think we can do better
<@bobius> OR or Object Restructure is our effort to do this
<@bobius> Some systems will be completely rewritten. Others will remain largely unchanged.
<@bobius> Now that the engine is available for all developers, we are focusing our attention to filling these goals.
<@bobius> This is why you will not see many feature updates to the Test Center
<@bobius> we understand that hacking is a problem
<@bobius> and that the lag and rubber-banding associated with the anti-hacking measures are a problem
<@bobius> so we are fixing those. After that, there will be no major updates to TC until it is time to test the new, restructured codebase.
<@bobius> We will now accept questions about what I just said.
<+Ellayn> Think of the current codebase as a giant, stuffed filing cabinet
<+Ellayn> Everyone has added things to the files, they're overstuffed and out of control. Everyone used different labels, different colors and different systems
<+Ellayn> So when you went to go find something in the cabinet, you could...but it took forever because of all the label systems.
<+Ellayn> The OR is us taking everything out of the filing cabinet, sorting through all the files and putting them into a new, clean cabinet.
<+Ellayn> Everything with the same labelling system, in the right order, where everyone can find it quickly.

<Yhor> My question "Are there any areas of the current test center that need our help with further testing? In regards to being able to carry it over to v1.."... as a follow up, is anything we are doing now helping with TC post OR, other than stress tests and "rubberbanding" algorithm fix implementation?
<@cRush> The Object Restructure will not be a patch to the current TC. It will be a noticeable difference at first; in fact, it will be a whole new species. Then it will quickly eclipse what you are all currently experiencing on TC, and race towards SWGEmu 1.0.
<@bobius> Basically, since a lot of the subsystems will be rewritten, they will all need to be tested again. As of now, enjoy your time on TC, but most of the bug reports will not help. We are keeping TC live so the community has something to do while you wait for the new codebase to be tested.
<+learningdisease> However, statistical information such as equations and density of spawns is useful to keep track of and report so that we can have an accurate pre-CU emulator
<@bobius> ^^ always true


<Coolcat> Can you explain 'object' so people understand how big and powerful 'object restructure' is?
<@bobius> evidentially my technobabble is confusing people
<@bobius> an object is just a way we organize the code. We try to mimic real-life things.
<@bobius> a puppy is a dog. a dog is a canine. a canine is a mammal
<@bobius> we got some of those connections wrong the first time
<@bobius> so we are fixing them
<@bobius> because no one likes puppies that meow

<Yhor>My question "Are there any areas of the current test center that need our help with further testing? In regards to being able to carry it over to v1.."... as a follow up, is anything we are doing now helping with TC post OR, other than stress tests and "rubberbanding" algorithm fix implementation?
<Yhor> "member:learningdisease" said"However, statistical information such as equations and density of spawns is useful to keep track of and report so that we can have an accurate pre-CU emulator".... My question is, where is the "Official" place we need to report these items of interest?
<@bobius> http://www.swgemu.com/bugs

<waltonsza>Why are the a lot of spawns that are in water and in the air ?
<@bobius> Good question.
<@bobius> The client you are using on your computer automatically puts you on the ground, and knows when you are in water.
<@bobius> NPCs and creatures obviously don't have this benefit.
<@bobius> We are working on implementing and fixing height and water maps, so that the NPCs will know where the ground and water is.

<Coolcat> Is it true that everything we can interact with is an 'object' within the database? Like character, mail, inventory, mobs, items, schematics?
<@bobius> yes
<@cRush> Yes, but there is more.
<@cRush> Objects entail everything from what you can see and interact with to things that you can't see. For example, a spawner object is not seen; events, skills, accounts, etc.
<@cRush> All of this is stored in a database
<@bobius> TC uses a type of database called MySQL to store this information.
<@bobius> OR uses a new system.
<@bobius> We've created a new ObjectManager that uses a database system called BerkleyDB
<@bobius> It will allow us to store an load objects much quicker
<@cRush> Resulting in less lag^
<@cRush> And more robust/accurate saves and loads.
<@bobius> a large reason you get LAG on TC is because it takes time to read and write to the MySQL server. The BerkleyDB is actually integrated into our server.
<Gargash> why do you feel that switching DB's will help the OR?
<@cRush> Summarized: Improved speed, efficiency, accuracy

<bobloblaw> it's a side issue but are there any type of things you would like potential devs to look at in the code? something that either doesn thave your time or something that would indicate an understanding of the concepts?
<@cRush> As always, we are actively looking for qualified individuals who have the ability to work in a fast paced environment and understand the application of many languages and technologies
<@cRush> While we won't hold your hand getting started, we do encourage anyone who believes they fit this description to lend a hand.
<@bobius> We all became involved with the project in the same way.
<@cRush> Those individuals who show that they can understand the overall flow of the project will be brought on board as an Official Developer, but that doesn't limit others from contributing.
<@bobius> Find a bug, fix it, send us your code.
<@cRush> #opendev is a great place to start if you feel that you are a qualified developer.
<@bobius> check out #opendev for help
<Likto> Is anyone working on getting droids implimated into teh game?
<@bobius> not currently. All active developers are working on restructuring the code for OR.
<@bobius> Additional features will come after we are finished.
<@cRush> While droids are included in the Object Restructure, they are currently tasked near the end goals, and OR may hit TC before droids are implemented.

<ShaydinTsepesh>for anyone interested : Will the OR make it easier to implement modern features like higher resolutions, and perhaps a z axis in the future?
<@bobius> No. These things are limited by the client that SOE has made.
<@bobius> Unfortunately, they have nothing to do with the server, which we are writing.
<@cRush> Client modifications are not entailed in the scope of this project.
<@bobius> Let me address the slew of Jedi and BH questions.
<@bobius> The current system that is in place on TC is NOT the system we will be releasing at 1.0
<@bobius> It was just something that was put in place to entertain the community while we are working on the OR.
<@cRush> It will be accurate to 14.1 Pre-CU
<@bobius> Spending time "fixing" a system that will not be part of the final project would be a waste of resources.
<@cRush> Controversial past features of Pre-CU, will most likely be toggleable in a configuration file by server administrators.
<@bobius> Stay on Target: We need to get this server finished. The quickest way for us to do that is to focus on the OR, which will be the final codebase. We need to abandon TC for a while, until the OR stuff is ready to be tested.
<@bobius> NO FEATURES WILL BE ADDED TO TC BEFORE OR

<@bobius> skitch: since the current tc be wiped can we please have the rubberbanding code turned of so we can actually play? speedhackers or no playing while we wait for OR is better than being stuck in a banding box for days...
<@bobius> The "rubberbanding code" is the anti-hacking code we discussed earlier.
<@bobius> This is one thing we are still working on for TC.
<@bobius> We need to have it running to collect data about how to fix it.
<@bobius> This anti-hack code will be the same code we use in OR, so we need to make sure it works.
<@bobius> While it is frustrating to be stuck in the invisible box like a mime, we are working as quickly as possible to fix the system, so it works as intended.

<@bobius> Ruledo: When will the public SVN repository be updated to that others can mod/work on it also (I'm assuming this would contain at least part of the OR)?
<@bobius> It is already public.
<@cRush> The current SVN is the current state of OR
<@cRush> There is much more in planning stages still

<torbersane> when the OR is complete will that allow for easier implementations of custom content? Will any options of custom content be done as modular so that one server could have a miner profession and say another server decided that it wants to be only pre cu 14.1?
<@bobius> Yes! A "Feature Manager" is in the works so that it will allow quick toggling of custom content
<@cRush> Many modifications like new professions will require client modifications as well as server side modifications.
<skitch> will the OR give the ability to add new races as playable?'
<@bobius> No, this is another thing that will require client size modifications. We do not currently endorce modifying the client.
<@cRush> Anything modifying the client will not be hosted, nor distributed via SWGEmu networks.
<@bobius> Are there any more questions regarding OR?

<kraken666> will we see any customer artwork in the form of transports, armor, housing, clothes, or any other player or dev generated items?
<@bobius> Adding new types of armor, buildings, clothes, or anything like that would require client modifications.
<@bobius> However, quests and events do not. We will make it possible for the community to create new quests and live events.
<@cRush> Anything involving new graphical artwork that isn't already in game is a client mod.
<@cRush> Reusing the same graphical artwork to create a "new" item is doable.

<Austen> Will the OR prevent slicing/lightsaber/armor/weapon issues that were on live which created lots of issues on server boundries that allowed players to do things that weren't intended? IE: Server boundries, will these be around or will there be a system in place to prevent this?
<@bobius> We use a different clustering system than SOE, so there will be no "server boundaries" per say. All loads will be equally distributed amount our servers. These problems should not happen with our server.

<MaelDiablo> So planets like Kashyyyk that SWG already have in their client can not be implemented by SWGEmu?
<@bobius> the client we are using is from patch 14.1. Kashyyyk was not apart of the client at that time.
<@bobius> Adding anything from a later client would require client modifications.
<@cRush> While it is feasible to add client modifications, we do not support this at SWGEmu.

<skitch> the OR will streamline how information is accessed and make things run smoother...is it possible that the information would be accessed too fast and cause bug to creep up like double stats being added to crafting and such because the information was in effect accessed too fast and applied twice?
<@bobius> It doesn't really work that way.
<@bobius> Old bugs will be fixed
<@bobius> new bugs will arise
<@bobius> That much is always certain.
<@bobius> It will be important to test all systems again once the OR hits TC

<Ruledo> So from a coder point of view, the OR is serialized data in the DB, and this means easier to use? Theoretically php pages can be made to modify the data in the DB? Is it possible to "hotload" this data and not require a server restart?
<@bobius> Very good question.
<@bobius> While BerkleyDB does not interface with PHP as easily and MySQL, there are libraries that allow it to do so.
<@cRush> There are also other options - like building a transaction server to handle remote administration
<@bobius> We will work out a solution to make sure there is an easy to use web interface.
<@cRush> This type of stuff will be discussed when we being building the Web-based administration tools near the end of the OR
<@cRush> Or rather, near the end of 1.0
<@bobius> "hotloading" content is possible, but some things will always require a server restart
<@cRush> The planned web-based administration panel, will also have public front where players can access things like their own stats, inventories, and guild information. Think WowArmory -> SWG style.
<@cRush> However, this is for further down the road

<skitch> Following Ruledo's ? will it be possible with the OR to have a webpage list your current skills, inventory etc that can be used for like advertising stuff your selling ingame, listing your skillset etc?
<@cRush> Read above^
<@bobius> All you PHP developers will have a lot to do towards the end of the 1.0 release

<torbersane> forgive me if this was already answered... with the OR will that allow for an efficient multi node server?
<@cRush> Clustered servers are inherent in our design.
<@bobius> Yes. We use a scripting system that is being designed in-house by Oru to insure efficient use of all servers in a cluster.
<@bobius> For you non tech types: We can use more than one server to increase performance

<@bobius> Coolcat: Could you add a 'LPE Timestamp' to player side files (forcing a tiny update at every connection), so the server knows who have been modifying client side code (like speed hackers) and kick them at log on? (sorry if it was answered before)
<@cRush> That means two, three, four, over 9000 computers to run TC -> improving speed, effficiency, and overall cap effectivelyt
<@cRush> That isn't how speedhacks work^
<@bobius> sadly it is not that easy.
<@bobius> Speedhacks modify the client in memory, they don't modify the executable itself.
<@bobius> The best way to defeat speedhacks is with efficient server-side checks
<&kyle> We will be focusing a lot on anti-hacking methods before release

<An0maly> The planned web-based administration panel, will also have public front where players can access things like their own stats, inventories, and guild information. Think WowArmory ==> won't that pose an attack vector to the server ? i.e : since you can access Character object directly over the web , won't it be vulnerable to code injection attacks ?
<@cRush> No.
<@bobius> We will design the system with security in mind. It is possible to make these things "read only". BerkleyDB does not use queries like mysql, so SQL injection is not a thing that would affect it.
<@cRush> Right, the objects fed to you will be readonly, pre-parsed objects
<@cRush> There will be no "direct" interaction with the server objects themselves
<@bobius> A few of us actually majored in Computer Security in college, so we will make sure your characters are safe
<@cRush> Others of us have been working in Web Development for decades :P
<@bobius> When our powers combine, we'll be more powerful than Captain Planet
<@cRush> ^

<Austen> Will the OR allow players to be dragged outside of Cells? Currently you cannot drag a player outside of a cell for some reason.
<@cRush> Currently we are investigating options for transitioning objects between cells and world
<@cRush> That requires a server side mapping of each individuals footprint
<@cRush> This sort of thing is a focus in the Object Restructure
<@cRush> We are also investigating improved Line of Sight (LOS) calculations.
<@cRush> These issues are all related, and part of the focus in OR, but may not be immediately handled.
<@bobius> The goal is to fix all bugs.

<skitch> for the us php developers out here what can we do to help towards the end of the OR...will there be an area for us to help you out? and should we start learning how php works with BerkleyDB to better help you out?
<@cRush> In the past, we have started a project called Packaged Community Portal, or PCP. It is basically a web based module that will cover everything a hopeful private server would need to cater to its player base and manage the server efficiently.
<@cRush> That project has been placed on hold while we try to get to 1.0. The reasoning behind this is that too much could change between now and then and it isn't worth having to restructure the PCP as well.
<@cRush> In the mean time, anyone who thinks they might eventualyl help on this project is encouraged to become familiar with OOP and certainly interaction with BerkelyDB may be a plus, but there will be other areas of work as well.
<@cRush> Until that time, though, I ask that everyone just kind of work on their own.
<@cRush> We will formally announce the PCP again when it is time to pickup development in that area.
<@bobius> Gargash: can i get an idea what specs would be required to run a server
<@cRush> That depends on many factors^
<@bobius> This will be hard to tell until we are nearing completion of 1.0. We will have to test the performance on different systems.
<@cRush> I can run the server on a 512mb ram win xp 1.2ghz p3 proc cpu...
<@cRush> It doesn't run well
<@cRush> It depends on many factors

<@bobius> Xavious: SO, with the OR, you are "formating" structure so that its all the same. that being said, with the code being opensource, would that not put the code back into the current slopiness it is in now?
<@cRush> And no hard, dependable numbers can be given until we are closer to finish.
<&kyle> Ai completion will be a large factor in evaluating needed hardware
<@bobius> That's what we're shooting to avoid.
<@cRush> We aren't "formatting" as in making everything a K&R coding style
<@cRush> It's much deeper than that type of thing
<@bobius> We have new procedures in place that allows us to peer-evaluate our code. Nothing will be committed to the SVN without making sure it meets our design standards.
<@cRush> of course we do require everything to be in K&R, and be readable with documentation, though.
<@bobius> I think we're going to start wrapping this up.
<@bobius> Thank you everyone for your questions

<@Serpentkaa> We are out of time everyone. Thank you for joining us in tonight’s DevChat and for the great questions. Logs of this chat will be posted to the forums in the next day or so.
<@cRush> Yes, thanks everyone. Great questions tonight! Sorry to anyone whose questions we didn't get to. They were flying fast!
<@cRush> Enjoy your time on TC!
<@cRush> Big props tonight to Serpentkaa, Ellayn, and Odwill who was helping us keep up with all the questions tonight! Couldn't have done it without them!

[Hamilcar]

Thank you for posting odwill and thanks to the devs for the informative responses.

[Serpentkaa]

Thank you Od for posting this!

[geforce]

TY for post

[Vlada]

Nice.

[Garrion]

Thanks, I wanted to see what I had missed last night.

[Omegataco]

That's one large nugget of info in terms of the OR.

[zetlaux]

Thanks!

[Labyrinth]

Awesome information.

And Bobius is just full of gems.

<@bobius> because no one likes puppies that meow

<@bobius> When our powers combine, we'll be more powerful than Captain Planet

[GarDavis]

encouraging

[TuleenDonai]

I'm still a bit confused about the Berkely DB structure vs. MySQL DB.

In looking at the code, it seems to me that you are using the Berkely DB for internal object management, AND you are using MySQL to manage player information that would get loaded and unloaded as players log on and off. That's where I get a bit caught around the axle. If I understand correctly, it appears that the Berkely DB would actually manage "In RAM" volatile information, but if the power gets pulled, you'll still need something like MySQL to maintain "Non-volatile" game state information.

This could be the result of an incomplete OR and/or observing "in-progress" code. There seems to be an overall thinking that Berkely DB will replace MySQL.

Are we to see future Berlkey DB constructs that actually "write" information into a file-based DB structure?

Can one of you Devs enlighten here?

[bobius]

Originally Posted by TuleenDonai

I'm still a bit confused about the Berkely DB structure vs. MySQL DB.

In looking at the code, it seems to me that you are using the Berkely DB for internal object management, AND you are using MySQL to manage player information that would get loaded and unloaded as players log on and off. That's where I get a bit caught around the axle. If I understand correctly, it appears that the Berkely DB would actually manage "In RAM" volatile information, but if the power gets pulled, you'll still need something like MySQL to maintain "Non-volatile" game state information.

This could be the result of an incomplete OR and/or observing "in-progress" code. There seems to be an overall thinking that Berkely DB will replace MySQL.

Are we to see future Berlkey DB constructs that actually "write" information into a file-based DB structure?

Can one of you Devs enlighten here?

Basically we will be using MySQL for two things (as of now, this may change).

1. Populating the static object db the first time you start the server after a wipe. (You can't load objects from the objectDB that aren't there)

2. Referencing objects that are in the berkelyDB

For example, when you login we hit the MySQL db to fetch a list of your characters and their objectids. We send those to you. You request character Tuleen which has an object id of 100001. We load player 10001 from objectdb.

Since all the objectdata is serialized, there's no quick way to poll the objectdb for information (like player object ids for your account), so we use myself for that sort of thing.

Only a very small amount of data is ever going to be read or written to MySql. That's why you get the performance increase.

[Ketaris]

Ditto to Uli and Kayliaah on the client modifications. I was a bit concerned when I read some of that as well. Can it be elaborated on a bit more? Has this always been the policy, or is it a recent change? Either way, what is the rationale? Does it conflict with the privacy policy?

[Kyrm347]

Thank you Devs for this Devchat, I appreciated the answers you provided for questions from us.

Originally Posted by Ketaris

Ditto to Uli and Kayliaah on the client modifications. I was a bit concerned when I read some of that as well. Can it be elaborated on a bit more? Has this always been the policy, or is it a recent change? Either way, what is the rationale? Does it conflict with the privacy policy?

I also was curious about the statements by Devs dring the DevChat about no new client side mods. This would cancel any future "upgrades" to much of the game. There are tons I of client side mods I think should eventually be added to SWGEmu that would enhance the game experience, some of which I think should have been ingame from SWG release.

My first instinct when I saw it in DevChat was that this is a "fine print" issue. If so I can understand why more has not been said about it.

Please if possible Devs, more information on this subject is desired by some of us. If necessary you can PM me with the info if you are reluctant to share it publically here, as I am seriously considering hosting and running a Server for SWGEmu v1.0. My future abilities as a Admin to modify and add content is an important factor in my decision making process as to whether I will go ahead with this idea.

Thank You.

[vformrh]

Missed the live devchat, but I'm glad this was posted so I could catch up. Thanks Odwill and everyone else!