They can't fully control the model, it does stuff even with instructions not to.
And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...
I'm not sure it's a job that you can win at even if you tried / were given all the resources.
This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
It's a broad statement and can mean different things depending where you are in that chain.
You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly.
Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it.
Then there is alignment, which is a higher longer goal.
Your comment about security questions is a matter of ethics as well.
If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
> is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no,
I wouldn't be so sure, most people I've met seem to be all too eager to suggest what worked for them (or they have heard is supposed to work) in conversations about medical issues, so I can't really see the public generally categorizing advice or guidance being given in non-professional setting or by non-experts as unethical. The taxpayers representatives having to bear the costs of the outcomes (at the very least as reductions of tax revenue) I can see being of a different opinion though.
Considering we've seen that you can't prevent models from leaking out bad things, I don't think you really answered my question by making a list of bad things.
Which is why you have accountability, transparency and auditing. In the same way humans make mistakes, this allows you to better catch issues, or at the very least diagnose/limit.
They can't fully control the model, it does stuff even with instructions not to.
And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ...
I'm not sure it's a job that you can win at even if you tried / were given all the resources.
AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w
There's no safety to be found.