Bloglines - Blue Blog

Bloglines user adam.saltiel has sent this item to you.



  Recent

Blue Blog

By Rat Outzipape

And finally a comment here:- The Blue Blog.

Comment by Adam Saltiel on March 15, 2010 at 1:41 am

I have read the full document, but not the referenced reports.
The big issue is how to reduce the big contractors hold on the budget.
I left my last job because I couldn't put up with the wastage intrinsic to our activities. IT can be very complex, this is true, but, all too often, the complexity is used as a cover for very expensive, very poor solutions. I am convinced that the real value of what I was working on was between 1/10 – 1/20th of what was being charged, hundreds of thousands, not tens of millions. This is a scandal. A scandal countenanced by HMRC, who are the client! I think it very likely that these savings are there to be made across the spectrum of government IT spend, amounting to billions that could be reclaimed every year. But, unfortunately, this is where it gets very complex. HMRC had a team of about eight, not all full time to this, overseeing a spend of >£35mil per annum. This is not feasible.
The complexity in this is
1. that Whitehall was stripped down under Thatcher,
2. it is likely to be stripped down again and
3. when this happens talent is lost. (And impropriety can creep in, I saw some evidence of this, too.) The first step must be to reinstate a sense of rectitude and public service. But to do that the Civil Service must be backed up, so that they can disengage from the big suppliers by wresting back control of those contracts.
There is much more that must be done in this area, this would be the tone that must be set.
Open source and a skunk works may help, but are not enough in and of themselves. Openness about contractual arrangements – if that is what is meant – could also help.
I expect there is a list of helpful items.
One key point is that there does not seem to be independent auditing. At the moment as the auditors are just arms of the IT suppliers. That successive governments have allowed this situation to arise is also a disgrace.
Where I was working recently was central to government policy, and I saw two things. That KPIs were redefined, in other words delivery was not made, in fact could not be made because of fundamental shortcomings in execution. That these failures in execution were blamed on all other parties apart from this supplier.
I also learnt, for example, that to change a URL, a simple operation of no more than a days work, would cost £100,000 when requested of a 'cooperating' other supplier.
That difficulties and failures I could see in our own system were not to be fixed as that was 'out of scope'.
And that the shoddy work I was involved with, that used out of date open source software that was actually costing money and resources just to keep in place, actually had no means by which it could be migrated thus entirely militating against any possible benefit.
I go into further detail about these issues and the difficulty in resolving them on my blog.

At least I have the chance to correct the howling spelling mistake here.


Bloglines user adam.saltiel has sent this item to you.




  Recent

Ctpr - Centre for Technology Policy Research

By Rat Outzipape

And further comments here ctpr after this entry better governance of public sector IT

Adam Saltiel says: 04/03/2010 at 17:16

  1. I am very surprised to find that there are, apparently, no Universities such as LSE, who have a strong interest in economics and economic policy that already research into this complex, but exceedingly important, area. Is this in fact the case? Could you provide references to any other such research or interested institutions including the National School of Government? A cursory look shows that there is a National School of Government strategy research project.
  2. [1.] shows the ambiguity in your use of the .org domain. It seems to me that yours is a commercial site offering paid for analysis and other services. This subject is very important and it behoves any organisation encouraging debate to be scrupulous. Without open references to other similar resources, cross references to parallel work, openness about your contacts in the major political parties and an open discussion of your immediate future it is impossible for an external party such as myself to know how or whether to align with your intentions. I would appreciate detailed clarification of these points.

[1.] From http://en.wikipedia.org/wiki/.org The org domain was one of the original top-level domains [sic]. It was originally intended for non-profit organizations or organizations of a non-commercial character that did not meet the requirements for other gTLDs. … Example use In addition to its wide use in charitable fields, it is often preferred by the free software movement, as opposed to the com domains used mostly by for-profit companies. Many political parties and support groups also use org domains.

Adam Saltiel says: 04/03/2010 at 18:10

  1. Much of the editorial and many of the comments on this site and on ideal gov cover similar ground and echo one another. While I have my reservations about these sites I find myself in broad agreement with these comments. For instance, looking at the top of this page ' … lack of a strategic relationship between IT and the UK's public services and public policy, combined with the lack of clear ownership … '.
  2. It seems to me that paradoxes enter the vision of very large spend projects. A constrained project in the private sector has a well delineated beginning, middle and end. At end the supplier must find other work, elsewhere. And always in competition with other suppliers.
  3. If there is a big government spend, whether through one supplier vehicle or through many competitors the delineation is not as clear.
    3.1. If the spend is with multiple small(er) contractors and there is a domino of failures, the one client becomes responsible. And if there is a domino of badly interpreted requirements this maybe more difficult to unwind between multiple suppliers. Although each one would 'feel' responsible, i.e. do their best to be proactive as their company would be on the line.
    3.2. If the spend is through single supplier it is very difficult for anyone to 'feel' responsible, because, in some way, no one is. This is why performance related pay is so important. Unfortunately the motive to increase company profit can mitigate against the discipline of performance related pay, and others are, as noted, demotivated anyway.
    3.3. The issue is whether the overhead of going for smaller suppliers unsustainable? Note that many comments advocate a smaller Whitehall.
  4. I think that from the outside we grasp at easy visions, such as the Civil Service is bloated, or they are naive and so on. I doubt it is a simple as that at all. But I also suspect that there is ground to be made up in the contractual arrangements made between government and their suppliers, itself an area for legitimate research and innovation.
    4.1. The software supply market to the government sector is very young and, as yet, immature.
    4.2. As commented by myself and others elsewhere, government has skewed the market. It has also skewed its maturation.
    4.3. Government do need a strategic approach to the market in order to redirect it. Government need a vision of what the market should look like and, I presume, that vision should be based on some good theory and not a few figures.
    4.4. Without either the theory or figures to hand my guess is that government needs to encourage the development of a few different types of suppliers, as well as to handle more (much more?) with small business.
    4.5. In building construction someone invented the idea of the concrete mixer lorry. I expect that now there are a few major suppliers, but no real monopoly in the area. This has lead to innovation, site mixing towers, the ability to supplier different and new forms of mix with little lag between lab and market and so on.
    4.5.1. Software seems different. Here suppliers wax and wan. Must this be so? Unlike my construction example it seems that suppliers in software rarely stick to a narrow specialist part of the market. Possible as a product of the complexity of the field? Perhaps government should be encouraging this?
    4.6. As I have commented elsewhere, there seems no equivalent of the independent partnership e.g. of surveyors, but for software, this model should also be valuable in other specialisms. In other words, divide up the specialisms and size them accordingly. Maybe this could be applied to legal services as well, after all why not?

    jerry says: 05/03/2010 at 08:20

Thanks Adam

Actually the LSE does good work in this area along with others – see for example "Digital Era Governance" (Patrick Dunleavy et al), Oxford, 2008. They also run a series of free seminars and events looking at a whole variety of related issues.

CTPR does a lot of pro bono and voluntary work – and, where we can find sponsors/advertisers (such as for our newsletters) we make these available free of charge. Where we do charge for reports, we do so to repay our associates and third parties who put together the research and reports as we run a co-operative model. Hopefully over time the paid-for research work will enable us to produce a greater volume of "public good", free reports. However, we do want to remain non-partisan and independent – so any advertising or sponsorship is unacceptable if it comes with "strings attached".

Personally, I have contacts and discussions with all of the major parties and aim to provide a balanced view of their intentions and capabilities with regard to technology policy. Our newsletter aims over time to have interviews with key figures across the political spectrum – our two most recent have been with the Conservatives and the Lib Dems. We hope the next one will be with Labour, although we also need to be sensitive over timing and the upcoming general election.

Adam Saltiel says: 05/03/2010 at 19:18

Jerry, Thank you very much for your response. I have now read the Feb Newsletter. I am impressed. The first section 'Trust in our digital lives' is a vital discussion, necessary for CTPR to cover, but I am restricting myself to cost saving in software delivery. In this the second article, the interview with Chris Huhne and the third, Where next, are highly relevant.
I have no background in politics, management or economics.
I have not been reading up on this subject prior to my own thoughts.
It is, therefore, very striking that nearly everything that Chris Huhne says I agree with, especially with respect to the need for the Civil Service to be empowered and the comparison with its historical self.
I must conclude these are fairly accepted notions by this stage that flow naturally out of any serious thought about the subject.
My immediate background has been as an employee of one of the big suppliers into government, though not particularly big in IT per ce.
I left this role because I was ashamed of the wastage and exceedingly low standard of work I saw.
I felt that if there was wastage that at least the standard of work should be high, that the work should be innovative, which is, after all, risky.
This was not the case.
I will not go into further specifics, however, I came to think about the nature of the way money was spent in the contract and the relationship between my employer and HMRC.
I realised that what ever the failings on the part of my employer they were being witnessed by HMRC.
I came to the conclusion that this cannot be an isolated case of mismanagement leading to very poor use of resources but was probably indicative of the whole software supply area.
As I could see projects that were being charged out at £10,000,000.00 that I knew, in terms of features, should be costing no more than £500,000.00 I have further concluded that, in software delivery, savings can be made in the order of 90%.
So I am with Chris Huhne when he calls for a more hard headed approach from government.
One of the issues government has is that it pre-announces how much money is in the pot.
This weakens (to say the least!) it's ability to negotiate.
I don't know the answer to this, but it does seem an answer is still needed.
Referring back to HMRC, I do find it astonishing that they allow a large proportion of the work done to be on time and materials.
My knowledge of this is very limited.
Forty years ago, in construction, any contract on time and materials was considered a gold mine.
Also, forty years ago was about the time when contract management was introduced and the independent management team consider a project a failure if they didn't run at least one of their subcontractors into bankruptcy.
(Achieved by refusing payment on the pretext of some gaff.)
Should government behave like this?
I'm not sure that they shouldn't. They certainly cannot when they vet their suppliers on the basis of 'due diligence'.
I don't know the answer to these sorts of issues.
I do know that the present state of affairs isn't sustainable.
I shall move on now to your review of the Conservatives.

My heart sunk when I read

'Is there an Open Source solution, saving development and licensing costs, and reducing dependence on long-term oligopoly suppliers?'
as this betrays ignorance and the desire to use catch phrases.
On its own, it is clutching at straws.

But I then read

'Where any bespoke computer code is written for the government, unless it genuinely pertains to national security, why can't it be released under open source licences?'
This, actually, is a very revolutionary suggestion.
Part of my shame and despair in my previous employment was that the vast majority of the code base was open source, there are two strong points to be made about this.

  1. Absolutely no contribution was made or attempted back into those open source projects. Frankly, that is a bit like theft, in the context or the mores of open source. It shows a glib contempt for the intellectual endeavour of others upon whom they depend.

  2. The code management and, therefore, the relationship with ongoing open source projects, was non-existent. Little or no benefit accrued from using open source.

It is excellent that the Conservatives should, I have to say very belatedly for main stream politics, suggest the use of open source, but this is not a thing to grab at, it is something that needs planning and management.
The second part, contributing code back, suggests the potential of the virtuous cycle may be appreciated. Again, this potentially has structural implications.
Some economic/managerial planning is needed to assess impact.
The final point I would like to make is about human behaviour.
It should be understood about mass psychology, that is any large group, that the very lowest common denominator is found in the group psychology and then a lot of mental effort goes into reinforcing the default positions.
The Conservatives use the phrase 'long-term oligopoly suppliers' wanting to reduce dependence.
But this dependence has been reinforced by a series of beliefs about safety of supply, and a number of other factors.
If there is a very large Ministerial demand the belief is that only a large supplier can fulfil.
When the Centre for Technology Policy Research says that the way public services are delivered needs to be reformed I am not sure what is being suggested.
The issue is should policy be top down led and, if so, must the accompanying technology be top down led.
These are two separate issues.
Large Ministerial demands do not have to be made and large fulfilment, all in one go, orders do not have to be worked up.
I suggest it is the later that should be the subject of research.
This area strongly intersects not just with economics but also with organisational behaviour patterns.
An area in which I believe the Civil Service is far ahead of the private sector.


Bloglines user adam.saltiel has sent this item to you.



  Recent

Everything Is Relative

By Rat Outzipadd

I have thought long and hard about this. I realise that I do, in fact, have to mention my employer and many of the details of what I have found wrong with the approach to IT being taken in the specific.
I shall explain why this is, starting first with my motivation.
As I point out in the title, everything is relative, but I think that people who are prepared to voice their opinions can, sometimes, make a slight difference to the way things are done.
The broad aim is to improve the way things are approached in this country so that -
1. things are done more efficiently in IT result:- increase in sense of purpose
2. consequent on 1. - money is saved. result:- more money for other things, allowing prioritisation of spending flexibility.
3. Systemic and structural changes are facilitated such that IT in this country has a chance of expanding creatively and unfettered by the growth of dark practices. result:- growth of intellectual capital in this area that can, as in a multiplier effect, be made use of in a flexible way across many different technological domains.

Reasons for mentioning my particular experience.
1. With concrete examples I can illustrate a few general points with some measure of conviction for the reader.
2. Mentioning one company name is far more likely to draw attention since there is an actual contract involved, parties to which having legitimate concerns.
3. My own words would soon melt into nothing and my ideas be forgotten by me before being expressed without the aid memoir of actual events to draw upon.
I consider this to be the missing link.
These arguments are well rehearsed and really devolve around the ideas of top down against bottom up.
The issue really is how is flexibility enshrined in legal arrangements. This is the area in which it seems to me government has shown the least understanding and total ineptness. Government repeatedly shows itself most happy when dealing in aggregate, as if commissioning large works such as battleships, hospital buildings with all ancillary services and so on.
It is clear that government in this country has not understood the significance of open source.
However, I cannot make that my plank here, it is a detailed argument for elsewhere.
What comes before a consideration of open source is the nature of software itself, and how best to reflect that nature in contract.
Here are some principals.

  • Software is never complete - so do not think of it as a product that is ever fully delivered.
  • Maintenance need not be end of cycle - an inflexible demarcation at this point means assessments have to be made about fitness for purpose, reassessed requirements and re-builds all focussed around a narrow point.
    Rather maintenance should meld into development phases which in turn should entail constant requirements reassessment.

So how can an activity that has, by definition, incomplete requirements and an ill defined end be held in a contract?

  • Key word - flexibility.

The contract will hold measures of flexibility. This means that requirements changes can be seen to have limited financial impact as against no requirements change in the same broad domain over the same period of time.

  • Key word - quality.

Quality has to be thought of differently. It is no longer just a case of testing against requirements and performance criteria. Quality is also a term applied to continual maintenance, that is readiness and ability to upgrade, as well as ability to substantially refactor (at every level from architecture through design to implementation), that is take out or substitute identifiable components.

  • key word - interchangeability.

This is another aspect of refactoring. A component may be taken out and replaced with another internal component or it may be replaced with an external component where a communication channel exists between the internal and external services.

All of these issues are well thought out and worked through, including the benefits of open source as well as service oriented solutions. There is no question that the approach needed to best capture the benefits could be applied to most, if not all, IT projects.
It therefore, becomes crucial to see the extent to which this is not the case and to understand why a suitable structure for IT is not being applied.

I have said everything is relative - the waist of money that I have seen in the context of my work is small compared to the the overall spend on IT. The waist of money that I have direct experience of is tiny compared to that, and it is that very small base I am extrapolating from.
If I were making a claim of malfeasance I would be restricted to a small set of hopefully supported facts. This isn't my point.
My point is that bad practice is systemic and it really is in the hands of the customer, government, to do something about this which would ultimately be to the benefit of us all.


Bloglines user adam.saltiel has sent this item to you.



  Recent

Government It Solutions

By Rat Outzipape

I think I have written enough to begin to come up with some viable solutions in this area.
As s pre-amble I should say it seems that this topic generates very little traffic and there actually is no central place that holds the public's and politicians attention where the issues are aired.
That is not the Cabinet Office Blog another way, a truly insulting and pitiful effort, not the (ctpr blog) [http://ctpr.org/] and associated, certainly not the Conservatives, where the recent story sunk like a lead balloon.
This is disappointing, although not surprising.
When one reads the comments of politicians there is always a feeling that they are a bit detached from the issues they describe.
Frances Maude is repeating more or less what he has been told to say. But how do the Conservatives' propositions add up?
open source software and a skunk works, but neither of these things on their own actually says very much.
As I point out in adjacent comments, open source can, and often is, abused by IT contractors. Abused in that they return nothing to the projects they depend on and abused in that they do not keep abreast of this and competing technologies such that no real benefit is derived.
A skunk works, as described doesn't necessarily sound all that hopeful either. Are they there to knock out, well, to knock out what? Cheap and cheerful solutions, exemplar proofs of concept, small scale focused demonstrators? And it seems that the team will be hopelessly small if anything substantial is to be tackled. So where does this leave the advice that government must seek? The skunk works looks like a red herring to me in terms of tackling the real issues.
As I have said the real issues are those of cultural change and business, mainly legal, context.
Unless contracts can be refashioned nothing will happen.
Unless the collusion between government and their contractors ends, nothing will change.
I have just contacted the Conservative party with this message:-

I have specific details of an HMRC led IT project where the contractor seems to deliberately provides excessively expensive and sub standard functionality in the service. I would like the opportunity to gather my thoughts on paper about this in the knowledge that it will be considered and lessons shall be learned. To catch your attention my broader surmise is that savings down to from one tenth to one twentieth of current costs could be made in many areas, amounting to many billions of pounds every year. Can you provide me with a contact in this matter? Please note I have read your proposals on technology, I am offering far more detailed and actual practice based analysis than the high level offerings I have read so far.

It is awfully difficult to get any attention for this sort of thing. Perhaps I just sound like a nutter?
Of course, I have a sales pitch 'from one tenth to one twentieth of current costs'.

What do I really think is possible and how would it be brought about? [ctpr blog] (http://ctpr.org/) in the guise of idealgov and associated wiki have many suggestions, one central one is that an inner team of no more than 15 head count become responsible for IT.
With Governance I will concentrate on a few points. With further time I will work my way through the whole document.
This is a quote of their own, all these issues have been thought about and written about by various organisations in some detail.

CIO sets out the top 10 habits of successful CIOs as being to:

  • I. Focus relentlessly on their organisation's strategy and how IT can enable key objectives
  • II. Provide leadership that motivates the IT team to deliver exceptional performance
  • III. Manage change so that it seems welcoming rather than threatening
  • IV. Spot talent, hire it, then develop it further so it becomes even better
  • V. Communicate ideas to the business in ways that are credible and convincing
  • VI. Understand how building effective teams provides a platform for successful projects
  • VII. Build effective relationships with internal and external stakeholders
  • VIII. Manage expectations about what IT can and cannot achieve
  • IX. Identify and harness the enabling power of new technologies to give their organisation extra competitive edge
  • X. Acquire industry knowledge so they can contribute to senior management decision-making

The whole of the IdealGovernmentITStrategy as far I have got reading through, seems to want to offer a vision underpinned by a series of recommendations and guide lines. But it doesn't, cannot, offer an analysis of what is wrong at the moment and what actual pragmatic steps need to be made to rectify the situation, starting from here.
There are sections on the need for review, the need for independent advisers and so on, and all of this pertains to these issues. There are certainly also, in this section, detailed mentions of responsibility assignment.
However, I am still left with the feeling that the whole proposal, were it to be adopted and I think the Conservatives probably are paying it some attention through some means (person contact?), lacks bite. It is not substantive.
I can only hint at what would be a path to a more substantive solution, possible details are too great, and the fields of knowledge mainly outside of my own.
This brings me to my first point.
The company for which I worked, Serco PLC have one research arm and their research is in, very sensibly, the legal aspects of government wide tendering. From this the device novel contractual forms, based, e.g. on research into PFI, to the government customer.
Two points. 1. It must be immediately apparent how attractive this can be to the customer, since promised and wrapped saving seem guaranteed on a very large scale. Only a very large company is in the position to make this sort of service offering. 2. The government is no match for this concentration of technical legal knowledge. Government needs to strengthened in non IT technical matters. I do not see this in the IdealGovernment ITStrategy.
What I do see, and this alarms me, is a call for 'independent' advisers from the industry.
The problem here such people are not independent, they are usually tainted by the way in which they have acquired their knowledge. To put it another way, the mind set into which they have been trained by their experience prevents them from being effective in the role in which they are being invited to perform.
CIOs and the like, especially those with experience in the public sector, are the last people to whom a radical program should be entrusted.
This brings up the very real problem of who then should it be? It seems to me that there are not the right people at the moment.
If we look at what is wrong with the current available people we will be able to see what may be done about finding more appropriate people. I will look at the points from CIO above in order to do this.

  • I. Focus relentlessly on their organisation's strategy and how IT can enable key objectives

My direct experience of this is that top managers and CIO's do this.
Question - what type of organisation? If we are talking about a PLC the foremost and single most clear objective is share holder value.
Share holder value translates into value for pension funds, so, really, mainly, value for retirement funds for working people with longevity of service.
What I am pointing out, very simplistically, is that there is a accrual of wealth for the pension fund at the immediate expense of the general tax payer. Private sector top managers who work with the public sector will identify with the organisational motives of their employer and those motives are in conflict with those of the public sector. There are few who will be able to then work in the public sector and realign their mindset, because it would entail admitting the original conflict.
As the public sector colludes with its own suppliers - this is a topic in its own right dealt with elsewhere - ignoring the basic inherent conflict, most managers who cross over will have little motivation to acknowledge it, nor, in consequence to the collusion, will they find the means by which they can make that acknowledgement concrete.
Now, what I anticipate from both sides at this point, faced with evidence of the massive mismanagement of IT, there will be an avalanche of self justification and rationalisation.
I very much fear that (ctpr blog) [http://ctpr.org/] and IdealGovernmentITStrategy, not to mention the BCS and CIO sets fit all too well into that mould.

There is a mountain of evidence that suggests this.

  1. The very long history of failures and lack of comment and action about this in public sector IT provision.
  2. The butter and parsnips taste of documents such as IdealGovernmentITStrategy, noticing that there is a fee seeking company behind it run by CIO types.
  3. The complete lack of science in the recommendations that are being circulated at the moment. To be serious for a moment, I think that there are several billion that could be saved every year in IT, but even if the figure is only, say, one billion, surely that saving is worth a lot to pursue? What, in fact, we have, is a situation where, as far as I know, almost no money is being spent on pursuing these savings, that is researching organisational structure, process change, economic ramifications and legal implications necessary to bring about these savings.
    There should be a dedicated group led by top lawyers and economists, whom I think may have a chance of being neutral, researching all these measures.
    Beyond that it is entirely naive to think that the interests of the client are reflected in the behaviour of service providers, as I will show.

II. Provide leadership that motivates the IT team to deliver exceptional performance

Serco PLC fail on this point and from here on I shall just mark this list pass or fail.
The primary motive of managers is in making profit in the profit centre for which they are responsible. I do not think that Serco PLC can be different to any other supplier in this regard. This can lead to extraordinary distortions. Exceptional becomes redefined immediately as that which is in the short term interests of the company and this will often mean towing the line for the project in hand no matter what opportunity there is for improvement. By definition, this must mean performance is less than exceptional by any common definition.

  • III. Manage change so that it seems welcoming rather than threatening

Fail. This company has been terrified of change and has not welcomed it in any form.
The ideal in the stricture parts company from the reality of daily life. Improvement and change, just doing the job right to specification and following the KPIs is seen as an organisational threat. The vested interests in continuing to do thing as is are so deeply entrenched, the sense that this is where the money stream is so embedded in the relationship between HMRC and Serco, that there is no prospect of change at all. HMRC have no plans to change how delivery is made, Serco have no plans to alter mode of delivery, and it is easy to see why.
Along with the issues of collusion, there is the issue of how money is made and that is on the basis of a very poorly thought out melding of the need to spend a set amount on the part of the customer and the need to show high costs (whether spread thinly or thickly across a cost base) on top of which profit is made. In other words the motivation built into this arrangement is to spend high, not to spend wisely. This is the most incredible disgrace, however I am sure it is common practice in government IT contracts.

  • IV. Spot talent, hire it, then develop it further so it becomes even better
    Fail. Obviously there is no motivation to chose talent as there is not outlet for that talent. I see no reason to believe this is not also common across the sector.

  • V. Communicate ideas to the business in ways that are credible and convincing
    Pass. But we already know that the business is not aligned in the least with its customer. What actually happens is that sub-project are proposed that disguise the short comings of projects that have gone before them, further degrading the system, often with devastating consequences.
    Is this common?
    Huge cost overruns indicate to me that it must be.

  • VI. Understand how building effective teams provides a platform for successful projects
    Pass, but see above.

  • VII. Build effective relationships with internal and external stakeholders
    Fail. Serco singularly failed in three major areas to build such relationships, each one has been pivotal to the project.

  • Failed in building effective relationships with other government departments and their suppliers in such a way so as to facilitate data sharing transactions between different sites. This negates the purpose of the Varney report (and has other very far reaching consequences) on which the whole exercise is predicated.

  • Failed to build an effective relationship with their third party IT supplier, BT, which has proven extremely damaging for the project as the IT core engine suffers an expensive and irreversible decline in reliability, performance and, above all, ability to adjust to changing needs.

  • Mismanage their own internal resources to the point of not knowing, by very many hundreds of thousands of pounds, the true state of their profitability. Thus expensively and misleadingly invite their customer to pay for change processes that couldn't be delivered on and were, in fact, nothing other than paper exercises that actually benefited their third party supplier rather than themselves or their customer.
    Since many other IT suppliers are on the other side of these reciprocal failures it is a fair conclusion that the failure is theirs, too, and that in conducting their business in this way they cause multiple failures of like nature with other parties.

  • VIII. Manage expectations about what IT can and cannot achieve
    Fail. CIOs had no idea what could be achieved with IT, it was, therefore, impossible to manage such expectations on the part of non IT managers. Moreover, as the technical team had little or no control over technical delivery they had very little credibility in their attempts to manage expectations, which looked more like papering cracks than anything else.

  • IX. Identify and harness the enabling power of new technologies to give their organisation extra competitive edge
    Fail. Absolutely not. See above. Most technology bought in - the bulk of the new technology on offer - came with its own unforeseen problems greater than the problems original set to be solved.

  • X. Acquire industry knowledge so they can contribute to senior management decision-making
    Fail. They seemed uninterested in industry developments in many key areas of which I am aware. One must understand that anything that costs money and seems unpredictable by virtue of being unknown is absolutely shunned. This can be understood in the light of my comments above.
    I could go on, but the point is made.
    Far from these strictures the reality is found wanting in a way that makes it difficult to imagine these points could ever be followed in the real.


Bloglines user adam.saltiel has sent this item to you.



  Recent

Jonathan Dimbleby

By Rat Outzipape

This is the text of an email sent into today's (27/03/2010) Any Answers. I have corrected to typos apart from any in the quote - for which I am not responsible, of course.

Cutting costs:- the intelligent way

I have these figures from the Adam Smith Institute.

Great Britain has gross public debt of $1tn, unfunded pension obligations to civil servants of more than £6.5tn, similar obligations to all UK residents of £4.3tn and "off-the-books" PFI obligations of $700bn: all up more than 8.9x GDP, before uncounted "off-the-books" healthcare obligations. *

But Britain admits to borrowings of only 72% of GDP.

Politicians find it impossible to face the real issues. Governments (the USA, Britain) have encouraged spending and loans to risky borrowers. While they vilify the banking industry they do not face their own part in this over the last half century. Politicians like quick gains that end up costing us dear.

In my own field of software engineering I have direct experience of money squandered in failing IT on a make work basis. I have had direct contact with millions of pounds being wasted. My estimate is that in many areas costs of IT provision could be less than one tenth. This could amount to billions every year. But government doesn't give Civil Servants teeth to manage these contracts. Teeth means specialised back up that studies the issues (lawyers, economists but not those tainted by the industry) and skilled teams of Civil Servants. This has implications to how the impending cuts are to be made. Squandering money makes shamefully bad engineering decisions that mount up future problems and costs, again for us the tax payers.

Adam Saltiel

*"Cameron's bank levy". To be published


Bloglines

user adam.saltiel has sent this item to you.







 
Recent


Ben Goldacre

By Rat Outzipape

This is an email to Ben Goldacre.
It is necessary to follow this thread through to see what happened.
Hello Ben, Some time ago, when working for the company, Serco PLC, that holds the contract to HMRC for businesslink, I noticed at the bottom of an email a quote from yourself. I cannot find it but the effect was that the businesslink site seems promising and that you thought people should continue to monitor it to see if it did live up to its promise. I am in the difficult position of having worked for Serco PLC on the businesslink account in a technical capacity and having formed very different views. I have a perspective, backed up by my own experience, on the quality of fulfilment in this contract that has a bearing on public IT policy. I think that my views could help inform, in small part, any debate. My principal concerns are 1. the enormous wastage and the corrosive side effect this has by failing to nurture talent 2. the stranglehold of suppliers that government has unwittingly handed to the few large contractors that is anti-competitive and forswears the benefits of a market system 3. the apparent lack of will and means by which government could successfully wrest back control of current contracts and let future contracts on a sound footing so that, for instance, pivotal KPIs are not rewritten by the supplier as they have been in the case of the businesslink contract.

I think that substantial savings are there to be made in software procurement, in some cases in the order of greater than ten fold - a huge claim on my part! But resources would have to be devoted in a persistent and concentrated way to make these savings. With respect to the savings, I hear government and the opposition making promises of savings in this area, but notice that the savings are predicated, more or less, on continuing the same pattern of procurement and project management. Further, if a contract such as Serco's with HMRC, cannot be examined for its failings, then how can any lessons be learnt? Clearly, were it to come down to it, vested interests would exert intense commercial pressure to protect those interests. The stakes could be high. However, my difficulty is that I have not been able to find the 'right' people with whom to express these views. I assume that what I have experience of has more general applicability than in the contract between Serco and HMRC, for this reason I am reluctant to take it up with either of those parties. I am not whistle blowing, exactly. Can you help me find a suitable avenue for these concerns and views?


Bloglines user adam.saltiel has sent this item to you.



  Recent

Ben Goldacre My Reply to His Reply

By Rat Outzipape

This one is long and laboured - but if anyone follows me beyond this point I have some further comments to make about this in a further post.

Hi Ben, Briefly I am 90% certain that it was not yourself. So I owe you a very sincere apology.

However, although you are not the person, can you help if interested?

Further, also if you are interested I continue with 1. an explanation 2. a story 3. a discussion below. Forgive the format. But emails are difficult, often intrusive and sent to people who may not have the time to read them. I'm just setting out what is below in case you are interested or want to be amused.

  1. This is an example of the footer in question.

...

Best Regards  Lorraine Carter| Senior Project Manager|businesslink.gov.uk  Businesslink.gov.uk, First floor, The Cottons Centre, Cottons Lane, London, SE1 2QG.   (For a map see http://tinyurl.com/SE12QG).    Land Line: TBA  Mobile: 07843 278600 

Want to know more about businesslink.gov.uk? - Read about us in the Guardian http://www.guardian.co.uk/technology/2008/sep/04/it.egovernment

Practical advice for business www.businesslink.gov.uk  The online channel for Business Link, delivered by Serco on behalf of HM Revenue & Customs 

I think that what has happened is that this was pointed out to me and I followed the link and read the article which is, more or less, as I paraphrased it, although more thoughtful. It is by Michael Cross, but it does mention in the last paragraph:-

All these factors seem to merit more external scrutiny than the service gets. It is possible that Businesslink is an example of Whitehall finally getting e-government right, but some independent confirmation would be nice.

I think it is possible that this paragraph and, in particular, the last sentence, has caused me to conflate the authorship with yourself. I don't read Cross's column, but I do read yours from time to time, with some pleasure, I should add.

I am 90% certain because on one occasion I took a copy of internal mail to me and have now searched through it. But I don't have all mail. The surprising and disturbing thing for me is that what I 'remembered' was a footer that said something like "Dr Ben Goldacre says 'What swung me was trying out businesslink.gov.uk, the government's all-purpose portal for transactions with business. It's one of the trio of megasites destined to take on nearly all official transactions in the name of joined-up public services and, in theory, lower costs.'

This, of course, reads like an endorsement. But, it seems, one made in my head!

I will try to check this again, just to satisfy myself.

  1. This is a story about a story about a ... . I have just read Ian McEwan's Solar. In it the anti hero, Michael Beard has a crisp incident on a train where he eats someone else's crisps thinking all the time and very indignantly that they are his, while the other person also continues to eat them with a strange degree of tolerance and forbearance. Beard then tells this story at a business conference, but is challenged afterwards to reveal where he heard and how he came to elaborate the story in this way as it is an urban myth, themed the Unwitting Thief. (This is a sub theme about the conflict between the literal and the metaphorical, but not fully developed by McEwan, I think.) However, I have been travelling back and forth to Berlin recently on a contract. Two weeks ago, disorientated as I was since I learnt the contract was ended, and no doubt tired from travel, I was in a Berlin fast food lunch shop and actually dived at someone else's tortilla chips, taking one and very nearly sinking it into the accompanying guacamole! For a moment, just for a moment, I thought the canteen had placed the chips on the counter for all to take while they waited for their order.

I was mortified, and as I sat next to the diner whose dish it was through my lunch, I repeatedly apologised.

  1. Discussion. It does seem that memory is very unreliable. But I don't think I had realised the extent to which it can distort things. This surprises me. All of this does connect with the original theme of the emails. The episode has helped me.

Is it really possible to save money in government it?

Skimming through the company emails just now I was reminded of the wealth of detail and the difficulty in decision making that went with my role. If there is so much detail, and so many people have to take part in a task which involves co-coordinating their time and their permissions to progress, how can costs be cut?

Technically, this must be the area of incentive management. It would have to be researched in relation to IT software development, but the evidence I have seems to point to the fact that the contractual structure between client and supplier and the incentives that flow from it have been very poorly designed on the part of the government.

While IT is complex there must also be some sort of law that states that a complex system will be presented as of greater complexity in proportion to the amount of money that can be found to solve that complexity.

Obvious really!

Best Regards,

Adam Saltiel


Bloglines user adam.saltiel has sent this item to you.




  Recent

Ben Goldacre My Comments on My Reply

By Rat Outzipape

One of this things that occurs to me in connection with this is just how easy it is to have a 'false' memory - which is what I think this is. My story also illustrates some other fragility of my mental apparatus. I attribute this to stress and disorientation, and not a little to shock. Behind there seems to be an over dose of tiredness and high blood pressure. So there are some health issues, some issues of managing myself and my time more productively and respecting that I get exhausted.
Turning to broader issues though. My false memory came at the same time that the story about Steven Byers and co. broke and it put me in mind to reassess what their motivation might be.
If we just read the press or listen to the news we really get the tail end of events, but thinking into their situation we find that there are two former ministers at the end of their parliamentary careers, who are being frozen out by the current party due to their own previous expressed opinions.
Now we do want politicians to have opinions, but the consequence of the wrong one at the wrong time can be dire for the individual.
This is the backdrop.

Given the above it is not surprising that Byers should be desperate, in his own shocked and disorientated way, to find what ever work and status he can lay his hands on.

The human factor is very great in considering behaviour of a system such as that of the relationship between government and IT suppliers. The real issue is can that collective behaviour be changed?

This is what this series of posts explores.


Bloglines user adam.saltiel has sent this item to you.






  Recent

Ben Goldacre His Reply

By Rat Outzipape

Ben's reply - which has some amusement value in his own footer!
thanks, concerned to hear my name being taken in vain, do you have a copy of this email footer? thanks

dr ben goldacre ben@goldacre.net http://www.badscience.net/

READ CAREFULLY. By reading this email, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. If you are anything other than a friend or an institutional professional colleague and you are writing to me about Bad Science stuff then it is reasonable to assume that I might quote our discussion in my writing, usually anonymously.


Bloglines user adam.saltiel has sent this item to you.



  Recent

August Jengler

By Hat Taglidbin

Standing in the bar, inevitably o my own, what is the difference between here and London? Well I guess few what we call black people, really anyone who is dark skinned on the whole, that is true or almost true, because this is the thing.
I forget where I am, I could be in London.
What is the difference in how people look here and how people look in London.
I love it here, there is something wonderful, but it is a major impediment not being able to speak German.
I think that the future lies here if people want a descent life. There is great potential. People are Eastern European, Slav, but that still doesn't quite distinguish them from how people might look at a bar in London. The point is that they are incredibly similar although I would say with a ruggedness in the men and a glory in the women not quite found in the manicured and groomed people at London bars. So, almost no difference apart from that there is the vast one of origin and the physical landscape to which a person is inevitably attached.
The difference that reveals itself in such small things, dress which is happily informal, expression, a different repertoire, adds up to a huge difference, a vast plane of differences stretching before me, encompassing not just Eastern Germany but the southern Baltic states.


Organising blog tags

conjoint.biz:1080
conjoint.biz:1080 tags
One thing I notice is that the cross wise linking of tags is easier to create in blogger in that I only have 26 posts so far. But there is still the problem of the tag's relevance. I notice that in the texai project he uses tags in an exemplary fashion. As would be expected from Steve Reed who worked seven years with Cycorp, Inc., the creators of the Cyc Knowledge Base.

Here are the categories Steve has settled on:-
* AGI
* Automatic Programming
* Cluster
* Construction Grammar
* Content
* Design
* Dialog
* Free Software
* Hierarchical Control System
* Java
* KB Implementation
* Lexicon Acquisition
* Linux
* Ontology
* Project
* RDF
* Sesame
* Skill Acquistion
* Spreading activation
* Technical Disclosure
* Web
* Workflow

That is 22 in all.
Notice that they are self consistent and that he has settled on a single style -
Capitalise, use spaces, no sub categories.
Actually he does have
Skill Acquistion
and
Spreading activation
rather than Spreading Activation. I cannot see the reason for that.
He also has a spelling mistake, the bane of any pre-thought tagging system. Though I would think he could correct that?
Steve does not discuss how he arrived at his categories, but it looks to me that he settled on them before he started writing the blog, perhaps possible where the blog has a very specific focus?
But I can say that it makes navigation of the site that much easier for the user.
This means that I am aware of the limitation of using tags in this site.
Further, on my other site (unreferenced here), there is chaos.
As to conjoint.biz:1080, well it is essentially a note book. I am not sure it it worth the effort for the benefit of the site - as opposed to my own benefit - to learn how to alter the url referenced on the fly. I have the problem that I can only create linkable urls that are within my firewall when on the home network, and I cannot do NAT to make them appear external. That means that if they are to be followed externally, I cannot follow them, which in turn means that they are not listed on those pages as referencing links, unless someone else follows them.

This suggests an alternative, which is to have a script crawl my site from my remote account, this should generate the links as it goes along.
This way I write all links as external. Of course I still cannot follow them myself.
I think that changing Nanoki is a bit of a mess as that would mean the code is bound to this web site and this port. It is bad enough that the site links are.
It does seem that at some point I need to dive into lua.
Not now though!

The rest of Boris comments in reply to my point 2. and 3.


2.
...
There's RDF support via Sail kindly contributed by Ian Holsman. And there's a serious (but interrupted) attempt to support XML Schema at the storage level with XSD types being synthesized with bytecode generation.
RDF support through Sail. I will have to look at this. I had thought Sail went through to OWL 2.0 but it must depend on the implementation.
XSD types .. so this must be dynamically generated, this would be better than the solution I found in Python using string interpolation and meta programming. That is if it is reliable. The byte code generation solution would be directly consumed in HGDB. It all shows how tricky and complex these things are since there have been countless implementations of Java consumers of XSD. I assume that once the XSD is represented in HSDB then any XML written to the schema can be read in. What about the strict non-strict verification issue? What about XML with no schema?
This also leads to another thought. As so much of the read write mechanisms exist in Python it is natural for me to ask if there are graph dbs implemented in python too.
So here is the result of that search:-
Looking at nosql-database.org there are some.
My main comments are.
1. Mikio Hirabayashi (Tokyo Tyrant etc.) is awe inspiring. And, also, it has a Lua API. If this proves anything it proves that my slight eclecticism is worth while, and leads me to blog next - as I anyway wanted - about another of my blogs.
2. It is strange that Sesame is not mentioned on this site. Anyways. Perhaps it is not NoSQL as the main way in is through SPARQL? Honestly, I have no idea. There are other omissions too, for same reason? But I notice Coherence is there as a grid database solution. Coherence is a distributed cache and this leads me to another thought that, obviously, which ever layer this solution is placed it may be trying to solve similar problems. This distributed cache is there to try to solve the problem of multiple calls from different servers through to one or more share RDBMS.
Graph Dbs store the data differently to the relational model in the first place. What I imagine is happening is that once the data is in the distributed cache the way it is treated is at least similar to the way a graph data base treats data. But that does not mean that the original data would have been better represented with a non-relational schema.
...
But the interesting part is in coming up with a representation and appropriate metadata that allows you to deal with dependencies across config files. This would be a challenging problem I think.
This is certainly a very sensible re-framing of my problem and I haven't given this any thought so far. It would be far better if I could describe my set of problems as a set of meta data types, such as server URLs, ports, version names and so on. This would avoid ad-hoc code and allow the far more interesting connections to be made that Boris suggests. I am extremely grateful to him for this suggestion.
Actually, when studying the maven 2 POM I did wonder if there was some way to classify segments of it. But I didn't pursue the thought. I believe I thought that I would not be able to find a correspondence between an Ant build file and the Maven pom, so I didn't try. But, thinking about it, this could be done. Here is the idea.
The Maven POM does have its own sections each with its own classification. This far I got and realised that for each section there should exist a set of rules, but I had no idea what the rules might be or how to derive them.
The problem here is that Ant exists using far smaller atoms - to use the term - than Maven. Each Ant atom is pretty much stand alone, a task that may have been defined and inserted at any point in the evolution of the build file. Now necessary for some aspect of the build when following that route. So how to map into maven? Is this a profile, non-default directories, property filtering or some other aspect? Given that maven prefers default configurations and also offers many phases built in, is the ant declaration necessary at all?
Well this is the idea. Go over these issues carefully, one by one and classify them, e.g. default directory structure, implicit lifecycle phase, property filtering.
Notice something about this list. The first two items are meta to the POM, but the last isn't, it is a direct reference to a POM element. So, it seems, that going over this very carefully it would be possible to derive meta descriptions against which arbitrary ant files elements could be mapped.
My intuition is that when ant files are written and added to it is the particular task and the ant build context that are held in mind, rather than any notion of the type of task being undertaken. That is the new task should complete when called in the context of the existing file. It is highly irrelevant that this task has been described a thousand times before in different contexts! Although ant is infinitely extensible, the normal set of tags is limited and indicates that classifying them should not be difficult. The above process should be two way, common ant tags can also be classified and then mapping rules applied that include a guarantee that the build context not be broken (how?) and whether the element should be kept, modified or deleted. One way to provide this guarantee might be to keep all elements and map them in (would that be possible? or not too messy?) and then removed as needed on successive iterations.

I can see that there would be some difficulty in programming this consistently without the guiding held of something like a graph db. A graph db invites the definition and use of properties and their conditional discovery. For instance the context of an ant task needs to be represented. It needs to be sufficiently well described to be able to make decisions about it. So although we know it is a type of task carried out a thousand times elsewhere, in this context what is special about it?
One of the things I have found is that ant builds perform many copy and create directory tasks because, unlike maven, there is no central repository. We need a classifier that can distinguish when this is the case. What other reasons might there be for copying artifacts from one location to another? There is the special case when it is necessary that the artifacts be the latest version in SVN. But this is exactly what maven is for.
There must be something like this:-
When all artifact dependencies are known, and the structure of the project (directory layout) is established all artifact copying, other directory creation and tasks to create the build path can be eliminated.
This isn't meant to be a complete description at this point, I am just pointing out that a lot of what is in the ant file would be reduced out in the maven POM.
I know that ant build files can harbour some exotic tasks. I think it is only with experience that it would become known how much of this process can be automated.
I should point out that the point of a tool like this is
a. to up lift the ant script to maven.
b. to facilitate ongoing changes in a controlled way, ensuring that where changes are made the consequences are understood and
c. that those changes are correctly propagated through the system.
Finally, to highlight, there are rules and there is process. Rules such as these could not be applied without a process that guarantees the new maven build is functionally equivalent to the old ant build.
I believe this is possible and is actually a desirable tool to have.
Boris has answered my points ...
What I have found is that the problem of converting from Ant to Maven is quite formidable. The problem lies in identifying correct jars. I will explain as I, at least, found it interesting, so perhaps others will?
Although the naming convention for jars in a Maven repo seems set actually it really is a convention that has unfolded over time. As it stands there are a few acceptable ways of naming - that is versioning - a jar, which any software would have to accommodate, but historically these few methods were not always followed. There are very many ways that a jar name may fall out of these few methods! And some jars appear in one place named in one way and another named another.
Perhaps this is a case for Collective Reconcilliation, I'm serious, it seems a good contender and so, HGDB would be very useful here.
I will explain a bit further. The information needed, if it exists, is in the few search engines that expose open source repositories and their meta data, the POMs. So this is a sue case for determining the unique identity of something from associated information. Just what information is needed would have to be researched.
I have mentioned the jar naming problem, one jar may have more than one name internal to a project as well as externally, coming from the time that naming conventions were not strictly adhered to.
There are two extreme cases that might occur here.
  1. No version information exists in the file name.
  2. The file no longer exists.
If no version information exists then it must be a case of comparing the content of candidate jars, and that, in itself, is difficult as the same class name can exist in different jars, often it is the same class in different version, but not always.
The file no longer exists, or never existed. Taking the latter first this would mean that an amalgam of code has been repackaged in the internal project, but, for some reason, it has now become necessary to find the origins of this code.
The other possibility is that the jar no longer exists, this happens when the project repackages their own code and, in the process, reversions existing artifacts.
I have encountered all three situations just mentioned, so these extreme cases may not be very rare!
I had not thought about, and did not then have time, to implement a generic solution to these issues at that point.

Finally I should mention that the means by which my code consumed the Maven XSD was through a string interpolation package and some hand crafting afterwards - I can no longer remember why this last step was necessary, but it was particularly hard. It seems that HGDB needs some similar means to convert XML and Schema into classes.

Knowledge Combinotronics
By this I mean a stream of different data feeds relevant to the user, each one manipulated by an appropriate algorithm to increase relevance and then offered for selection by the user. That is the user would select some combination of the presented data and that would then be used in the next phase. Really there is no magic going on but that user selections are amenable to an iterative process of selection and reselection in different or unfolding contexts.
There are many examples of how a context may be created depending on use scenario but one would be where  they are fed back into the data set for other users immediate guidance.
This depends on high throughput infrastructure, typically this has been expensive.
'different or unfolding contexts' this is key here.

posted by Kobrix Software at Kobrix Software, Official Blog - 3 weeks ago
NoSQL has picked up a lot of steam lately. HyperGraphDB being a NoSQL DB *par excellence*, we will be joining the upcomping conference organized by the 10gen, the maker of MongoDB: "NoSQL Live from Boston...
These are the comments I have made and the reply from Boris.
I gather them here for any further comment I may want to make.
See after the last comment.


3 comments:

semanticC said...
As its the 12th I guess the conference has taken place, links I will be following up. There are a few questions I have. 1. Can you point me towards any comparison of NeoDB and Hypergraphdb, do they cover the same ground? How do they differ? 2. The relationship between graph databases and 2.1. OWL, how would OWL be consumed, or would it? 2.2. more generally RDF, and then XML, after all there are XML databases that parse in the XML. How do they compare? I'm sure I have missed something(s), but what? 3. One of the problems I have encountered is in keeping various .properties files aligned. One approach is to use something like magic lenses such as the augeas implementation. But, at the same time, I have wanted to rewrite these properties out of their ANT context into a Maven POM context. A job for hypergraphdb? Ideas? 4. Moving on, I have noticed the fascinating post about using hypergraphdb to create a neural net. 4.1. Would you agree that what is happening here is in line with Rickard Öberg? http://www.qi4j.org/ for background and http://www.qi4j.org/qi4j/351.html where he discusses the relationship between algorithms and OOP. BTW, he also arrives at the need for atoms and mentions the same focus, the business case, that you emphasise in your background paper, Rapid Software Evolution. 4.2. I notice that Neo4J has an example of a spreading activation algorithm (token passing), http://wiki.github.com/tinkerpop/gremlin/pagerank - I expect this means that either db could also be used to implement Random Indexing - sparse matrices - as developed by P. Kanerva and M. Sahlgren Some of this may be touched on in the Disko project. Again, ideas? Sorry for such a long comment, but not sure how/if to email privately.
Kobrix Software said...
Hi semanticC, A good place to discuss HyperGraphDB would be the discussion forum: http://groups.google.com/group/hypergraphdb?hl=en This is a long list of topics raised indeed :) Let me try to cover them one by one, perhaps in separate responses: 1) Such comparison should ideally be done independently and I am not aware of any. For starters, HyperGraphDB has much more general data model than Neo. In fact, the name is maybe a bit misleading from a functionality perspective because now it's being labeled as "another graph database", which it is, but it is also OO database, a relational database (albeit nonsql) etc. In HyperGraphDB, edges point to an arbitrary number of things, including nodes and other edges Neo is a classical graph of nodes and directed edges between any two nodes. In addition, HGDB has a type system while Neo doesn't. So HGDB has in effect a dynamic schema that you can introspect, reason about and change. Besides the data models, the storage models are quite different: HyperGraphDB has a general two-layered architecture where a big part of the storage layout can be customized. Neo uses linked lists to store its graph and claims that this makes faster traversals (probably true) and that this is all you need to do with a graph, you don't need indices, pattern mining etc. (here, I disagree). HGDB relies heavily on a lot of indexing for more complicated graph-related queries & algorithms. In sum, HyperGraphDB has pretty much the most versatile data model I know of, and subsumes Neo and others easily. Weather that sort of generality comes at the expense of performance remains to be seen. As you've probably realized from the neural net post, HGDB gives you more representational choices so performance has to be measured more globally, at an application level, through a design that makes intelligent use of what HGDB has to offer. more on the others later....perhaps at the end I'll sum up my responses in a separate blog.
semanticC said...
Hi Boris, Thanks so much for your reply. It would be great if the other questions inspire a blog post. If anyone is interested the NoSQL conference is previewed and will be written up here http://radar.oreilly.com/2010/02/nosql-conference-coming-to-bos.html - and it is a good discussion. Boris contributes too! There are still many things I cannot get my head around. I can see the 'representational choices' the ability to define functions directly working on the data using the HGDB API. I expect this is a good thing in the way that, for example, annotations are better than XML, everything is in the place where it will be used, which facilitates concentrating on the task. But other benefits? Here I cannot see. Moving on again, I am reminded of the efforts of Henry Story to create a framework to import RDF, inspired by Active Record. I am very unclear about all of this. Did I read somewhere that there is a standardisation of the syntax for the import statements of RDF namespaces? Anyway, the idea would be to make the referenced ontology available in code, presumably it would already be in Sesame as the graph db backend? All of this seems relevant to HGDB. First you have mentioned the type system, so how to model the types? I had thought that OWL was a good way of both modelling and sharing those models. But if so, what of the other aspect of HGDB, its ability to deal with semi-structured data, how to fit the two together? I am thinking about Collective Entity Resolution as perhaps one sort of solution and simply in code, how they might interact, as another area. Moving up towards the goal of evolutionary software, I have long thought that it must be possible to describe software using OWL. I assumed that reasoning would take the place of a lot of code when there is a well constructed model. Of course that brings me back to what role reasoning in NoSQL. I know it is build in to AllegroGraph. As I say, many thoughts, but I don't really understand the ramifications of NoSQL at the moment. Perhaps I am missing the point altogether?
top