Our most painful experience of migrating to Open Source – trixbox softphone clients using Hudlite arent good enough

As previoulsy blogged, we completed our migration in full to open source over Xmas, and have been running everyting possible on open source for 5 weeks now.  Without doubt the biggest problem, and source of legitimate continual complaints from our staff, our customers, and anyone who used the system, was our trixbox phone instalation.  Negative symptoms included:

  • very poor call quality, so bad we were often told to “get out of the dustbin” and we sounded like daleks
  • Calls cut of in the middle of a conversation
  • Calls to a group were notifying to the SNOM hard phones in the group upto 10 seconds before the softphones, making it very difficult to identify calls
  • SNOM M3 DECT SIP phones didnt work, but SNOM 300 hard phones did. Sometimes.
  • Call handling using Hudlite was not integrated with Ekiga (or anything), and so it was very hard for users to forward calls around the office, and many, many calls were missed as there is no way to notify users of an incoming call if the ekiga box is minimised.
  • the system often had network problems, such that we replaced the ADSL router and moved the DHCP function several times in a month to get some sort of stability.  Last week alone, we lost about 6 working hours of the working week with no phone system (yes, no phones at all).

By the end of the 4th week, morale was pretty poor, and after a crisis meeting we decide to do 2 significant changes.

1) bring in Matt and Dave, both sys admin type people from our open source support consultancy Senokian on a “do not leave until you have fixed it” mission to find out what was wrong with our setup.

2) abandon the softphone system – hudlite and a SIP client (like ekiga) it is just not good enough for a business environment; we bought a new batch to SNOM 300 hard phones for all the previous softphone users.

Very satisfactorily, a major trixbox/network fault was found by the superb Senokian chaps, and fixed, and after that all the new hard phones worked perfectly.  That was friday.  After a very frustrating period, we should have a much more stable week this week, but have leant some lessons that others may wish to heed in due course.

  1. The Hudlite system built into trixbox is difficult to work, and we do not recommend it for busienss use.  It desperately needs an inbulit SIP client, and is almost unusable without such a  development.  We know this, becuase we used a SWYX VOIP phone system that was so integrated for 4 years before trixbox.
  2. If you have the same symptoms as us, then check your trixbox setup, so that it has the right local network subnet mask, or else, as we found, if you use an incorrect one (probaly the default), all the network traffic tries to go via your  network router (ADSL in our case), and it dies with the repeat NAT and repeat boucing of packets.  In our case, it was set as (i think) 127.0.0.1, and it should have been 192.x.x.x etc.

I hope this post helps someone.  I will report back after a few days on what we experienced this week, but it looks good.  Just dont make our mistake, and try to use Hudlite softphones for a business critical operation ie a working phone system.

It may be a conincidence with the snow and bad weather this week, but the disruption to our operations has been very significant, resulting in the one of the lowest level of sales against target in my memory.

This is a profit warning to businesses – unreliable phone systems hurt.

    • – unreliable phone systems hurt.

      More like phone systems installed by the inexperienced hurt.

      I provide remote support to Asterisk based systems and you would be amazed at the number of times people called in “we hired this guy to do our system and now we are stuck and it does not work and he doesn’t know how to fix it, or is unresponsive… Or has gone missing.

      The key to Asterisk based phone systems in business is to hire a professional to begin with.

      I almost started this rant with “If I had a nickel for every time…”

      But, we charge $150.00 an hour. So, I do a bit better then that.

    • That may be fair Robert, but its equally as likely to be harsh. The installer has proven himself to be more than competent as an open source sys admin on many occasions, and has done many trixbox instalations much bigger than ours.

      It maybe that we did it on too tight a budget, but with respect, I doubt that even trixbox experts, possibly such as yourself, ever get an instalation right 100% first time.

    • Chris

      How are these Trixbox issues? Sounds like poor planning and network issues to me. Hudlite is not a softphone, it is a desktop tool which is free, as in open source, you didn’t pay for it and you don’t have to use it.

      Also, it is obvious to me that this project was poorly planned and severely under budget. Why would ANYONE use a softphone as their primary means of communication. It is intended for remote access to system; I would never recommend or support it due to so many factors like clients PC which I have no control over.

      I may sound like a Trixbox fanboy, I’m not. I am pretty quiet guy, I usually laugh at blogs like these. I could not help my self but to respond to these asinine statements.

    • We make our living installing phone systems for people and we very often get it right the first time. This is because our business model is selling direct to the end used and through resellers. We don’t have the luxury of fixing problems onsite. Because we are never onsite. We ship the system and the end user or a reseller installs it. We have spent years coming up with a process that make sure everything goes right. For example we heavily test our trixbox PRO product with very specific versions of phone firmware. We also create the configurations files for the phones so we know they are configured right. All of this is with our per seat license product, trixbox PRO.

      For people who want to go it alone we make trixbox CE. The version you are using. This is very flexible and easy to use. In most cases it can be installed with not problems. But without knowing what version of firmware your using on your phones or how you configured your phones it’s easy for problems to happen. I would recommend reading our user forums. We have a huge community world wide that uses HUD and trixbox CE. There have been many posts on the problems with using soft phones and the importance of setting your network up correctly.

      As for HUD it is designed to be used with a hard phone or a soft phone and integrates very nicely with both of them. It works well with a mobile phone as well. I was in an airport the other day using HUD with my cell phone. I clicked on a member of my team. My cell phone rang and I was talking to him. If I only had a soft phone this would not have worked because the wireless broadband I was using is too slow to carry VoIP. The point of HUD is it decouples the voice from the control of the phone system and you can use any device even a pay phone as the voice connection. I can be completely productive in an airport just as if I was in the office. This is the power of HUD.

      I you are interested in working with us we can help you set up your trixbox to work as you want it too. It sounds like most of the problems you are having would be easy to resolve.

      Andrew Gillis
      Trixbox founder

    • Hi Chris, and thanks for your comments.

      This blog covers what its like to migrate to open source from an SME’s perspective. I am not an IT professional, I dont have the skills or ability to plan like one, but this is the reality for many smaller users. You do of course state the obvious that we dont have to use it, but we choose to, and are great fans of open source.

      I have to take issue with a few points:

      1) No, we didn’t pay directly for the software, but I assure you it has cost us a lot of cash to migrate in professional support, its indirect payment for the service and product. And when we got it wrong, it financially hurts us in downtime and loss of business.

      2) For 4 years we used a softphone (SWYX) across our business as our primary and only phone system. It worked. Well. There are many, many businesses that use softphones as their main phone system. If you dont agree with this, you dont know much about phone systems.

      3) the final fault was 100% a trixbox config file setting. Thats why it was a trixbox issues. Now it works very well.

      Insult the migration process as much as you like Chris, I’ll authorise all your comments. But, I’m afraid this is the practicality of doing things on a SME budget, we dont have the knowledge, skills, timescale or budget to do things at the enterprise level.

      Dont get me wrong, I really like trixbox and we are staying with it, but there are lessons here from others, at least I hope so!

      These posts are more “brutal truth” than asinine.

      regs

      adrian

    • Hi Andrew

      Thank you for visiting the blog, I appreciate the time for your comments as the founder of Trixbox.

      As of Friday afternoon, 4 days ago, our Troxbox system is working perfectly, but are now using only SNOM 300 / M3 hardphones, not softphones. No need for support to fix it, but thanks for the offer, much appreciated. AS you expect, we have been through the forums many times, and highly value them as a resource, as do Senokian.

      I dont actually see HUD being as useful to us as you describe, probably becasue for years we have grown used to a single, combined peice of software (SWYXIT) that does the dialing, call handling, switchboard etc, all in 1 go. Its this lack of integration thats so frustrating, For a normal office worker, used to a phone, its just too cumbersome. and I know this becuase we tried it, with training, and 5 weeks of serious efforts from my whole company to get to grips with it. In th end we gave up, and invested in SIP hard phones.

      If there are any plans to merge a SIP client and Hudlite, then I’d be really interested in this project, and will support it.

      Again, thanks for your comments Andrew, your product is great, even more now that it works perfectly for us!

      regs

      adrian

    • Greg Guldenschuh

      Ardian,
      I am a small company myself and last year made the move from regular phones to Trixbox with Aastra SIP phones. Now my budget was under $1,000USD and the final number was something like $700USD. I bought 2 SIP phones and 2 Linksys pap2t analog phone adapters to support our existing phones. I even installed a softphone on my PC to test and it is still configured into the system.

      I spent 2 full months testing the Trixbox and getting it to connect calls before bringing it online. Since doing so, I have made one minor configuration change to improve performance. I have NEVER had any bad call quality on ANY phone!

      Because I use my PC for running fairly compute and I/O intensive applications, I made a choice to not use a soft phone specifically to avoid the possibility that I would encounter the bad calls and “dalek” like voice quality.

      Now, I am an IT guy and a phone guy–and have been for more than 15 years. I hear people preaching the gospel of convergence and integration on a daily basis. IMHO, there are times to converge, and there are time to not converge. There are scenarios when integrating is called for and there are scenarios when integration should be avoided. The value that experience in both IT and telephony brings to the table is to be able to know distinguish between the different scenarios and know how to use various technologies so that they can actually work together, not get in each others way and cause the end users problems.

      Please do not take my observations as any type of comment about you, your vendor, or your Trixbox installation. I have no knowledge of any of them and cannot comment on any of those. I have found Trixbox to be quite useful in my small company. We transfer calls, park calls, call various extensions, and make many outbound calls every day. We have not encountered the issues you speak of.
      I chose to stay away from soft phones because

    • Hi Greg

      thanks for the comments. Sorry that the bottom got lost from some reason.

      Fair comments, I wish I had your level of understanding when we started this process! Actually our trixbox was configured offline in august 2008, tested in the autumn, and rolled out over the xmas break across the company. Yes, there was some big bang, but to be honest we dont have the ability to run a full scale trial without going online. Trixbox is great, its the softphone bit we got wrong, and that hurt, as well as the network traffic logjam and unrelaibility that was just not there during testing, it worked perfectly with only 1 or 2 users. Some things we only found out when we went live with 15+ users!

    • I think its unfair to say trixbox or any aspect of our phone system was installed by the “inexperienced” as clearly it was not. As mentioned, Senokian have experience installing, configuring and supporting asterisk/trixbox and so on. Initially it did seem like it was our network causing the issue, therefore doing tests on each section of the network, even replacing a router, using different switches, isolating areas were a must. We did these and found no problems with our network; it is set up and running as we expect, as a pretty standard network would run. We tweaked the network to see if it made any difference, which it did not, and at the end were left with the same setup as we initially had, using our first router.

      As the network was running fine, it meant the problem lay somewhere else, and after a few hours poking around files, the chaps at Senokian found the problem. The setting in one of these configuration files(I’m unsure exactly which one) had an ip address which it should not, either by default, or because it got the address itself from somewhere else! Senokian didn’t put this address in, nor did we; as we never would have needed to/wanted to/did.

      Poor planning – I wouldn’t go as far as that, we planned this migration of the phone system for some time, trialled and tested what we could, but due to limitations, probably the same limitations most SME’s would face in this scenario, we could not do a 100% test until we went live and put the system into the same capacity as it would be in a normal working day. We may have been able to do more testing, different situations and so on, but hindsight is a wonderful thing!

      The softphone aspect is hard to overcome, we wanted to use softphones from day one, have done for years using similar windows systems and programs, so wanted the same functionality. It may not suit other companies, but our company staff were used to softphones and to be honest, comments from the staff who use the phones also stated they preferred softphones. From testimonials, reading up online, forums, blogs, documentations and so on, we thought we would be able to get the same functionality as before, which has proved to be the big flaw in our plan really, resulting in hardphones.

      Looking back, we may have been able to do things differently, and perhaps could’ve avoided some of these issues, but I believe the problems we faced were not down to poor planning(we planned and tested everything we could), not down to our network (as proven), but a rogue config file which, when fixed, basically resolved all the problems. By this time we already had the new hardphones, so the softphones may now be ok, but rather than messing again, we have a functional system which has been 100% stable since the configuration file was altered.

    • It looks like there has been some further interest in this post, its been covered over at ZDNET http://blogs.zdnet.com/Greenfield/

      I’m not sure how we have come out of it, but its a lively discussion!

    • Rich, time to “man up” a bit here.

      The experience level here is fairly transparent. Telephony is a complex subject and a “sysadmin” however insanely great in some other arena, does not translate to time and experience installing ISO based Asterisk distributions.

      Time and experience, and I dare say exposure to this Asterisk community, perhaps would have caused everyone less grief. Every writeup I have encountered on installing trixbox indicates sip_nat.conf as something to configure correctly on install.

      The lesson here?

      Don’t ignore the common “sysadmin” procedure of testing and proceed w/fallback.

      I am just saying…

    • Pingback: book review - trixbox CE 2.6 by Kerry Garrison « Mercian Labels - business, open source & security label printing()